From Numbers to Names

From Numbers to Names

Artificial intelligence program can identify Holocaust photos

These Holocaust refugees, all orphans, arrived in Israel in 1944. Photos like this one can be uploaded, and the AI behind Numbers to Names could find images of their relatives. (Zoltan Kluger/National Photo Collection of Israel)
These Holocaust refugees, all orphans, arrived in Israel in 1944. Photos like this one can be uploaded, and the AI behind Numbers to Names could find images of their relatives. (Zoltan Kluger/National Photo Collection of Israel)

We’re used to seeing pictures taken of Jews before or during the Holocaust. Black and white photos, some grainy or ill-focused, others sharp and artful.

Some of the older pictures show hopeful-looking children or happy family; as time goes by, the expressions becoming increasingly grim. By the end, they’re desperate-looking faces, staring out from a mass of other faces.

We look at them, hope that they’re the pictures of the few who escaped, knowing that by definition most of them did not.

But in most cases we don’t know the names of the people in those photographs. If they’re not family heirlooms, preserved with names written on the back, the people in them became anonymous long ago, even as their faces haunt us.

But technology often makes the seemingly impossible —retrieving names from the fog of unrecorded history — possible.

Of course, it’s complicated.

One of the seemingly impossible tasks that artificial intelligence can perform is facial recognition. That is, it can identify a person from an image; it can put a name on a flicker in a video. It’s amazing.

It’s also controversial; it can erode privacy, even when it works right, and often it does not. So far, facial recognition applications tend to be far less successful with people of color than with white people, and that’s resulted in many false identifications. It’s particularly dangerous when it gives law enforcement agencies incorrect results.

It also, sometimes, maybe often, maybe with increasing accuracy, can give the nameless back their names.

Daniel Patt and Jed Limmer, friends who are undertaking this project together, “are trying to connect people who are alive today with photos of loved ones who they lost in the Holocaust” — people who were murdered, or people who survived but were lost to each other nonetheless in the chaos of the war.

The project allows people to upload their photos; From Numbers to Names, as Mr. Patt and Mr. Limmer call their project, will select 10 images from the archives to which is has access — mainly the archives of the United States Holocaust Memorial Museum, which has more  than 34,000 photos, many of them with dozens of faces — that are possible matches with the questioner’s uploaded images. Altogether, there are about 500,000 images that the project can search; that’s about two million faces, Mr. Limmer said.

The program is new; it’s still imperfect (and like all human-made projects likely will remain so, even after most of its kinks are worked out). But it uses new technology to help an age-old need to connect.

Mr. Limmer grew up in Closter; he lives in Haworth now. Mr. Patt grew up in Bedford Hills, in Westchester County; he lives in northern California now, but he has local connections. His grandparents lived in North Bergen.

Mr. Patt, who is a software engineer at Google, thought of the project when he was at the Polin Museum in Warsaw in 2016. “I was just overwhelmed by the feeling that I was looking at family photos,” he said. It made sense, because three of his four grandparents were Holocaust survivors. “I knew all of them,” he said; one, his 91-year-old grandmother, lives in Manhattan now.

“I had the thought that if I left the museum, I would never see them again” — those haunting photos, those faces, possibly of his lost family — Mr. Patt said. “I didn’t know if they were online.” So he started thinking about technology, quickly moving from how to digitize them and make them accessible to the larger problem of how to identify the people in them.

Daniel Patt, left, and Jed Limmer

He talked to his friend Mr. Limmer — the two men met more than a decade ago when they both worked at a bank, but today Mr. Limmer, whose ancestors left Europe before the Holocaust, is a wealth manager.

“Dan talked to me about this project,” Mr. Limmer said. “I’ve been on the board of a small nonprofit, so I have some experience in the nonprofit space. This idea is powerful, and I wanted to be involved in it.”

So together, working from different coasts, the two men brainstormed to put together From Numbers
to Names.

They’re working with a huge number of photos from many Holocaust archives, taken by many photographers, for a range of reasons. “There were really interesting photos taken by a Bulgarian photographer,” Mr. Limmer said. “He captured the Jews as they were being registered in Macedonia.” (Those photos, among the many that the project uses, are available online, along with many others on the United States Holocaust Memorial Museum’s website, Google Bulgaria for this set of terrifying, deeply sad images.)

Other photographs were taken by Nazis; some were “from the pre-Holocaust era, of people just living their lives,” Mr. Limmer said.

“I felt like when I was looking at those pictures, I was looking at myself,” Mr. Patt said. “These weren’t people who lived hundreds of years ago. This was 80 years ago. I felt like there was something wrong with not working to identify them and sharing them with their descendants.

“The project got started in earnest in the last few years, precovid,” he continued. “My original plan was that people could go to a museum, there would be a private booth, and you could be photographed there.” That photo would be uploaded immediately, “and then there would be a private experience, when the entire collection would be compared to your face, to see the pictures most similar to your own photo.”

Right now, not everyone who tries will find a face he or she recognizes, “but if you are someone who has lost a relative in the Holocaust, or if you have a connection, there is a reasonable chance” that you’ll be served that person’s photo. That reasonable chance “is the key point of the project. It’s the best effort to find a photo of your family, but even if you don’t find it, it’s still worth trying.”

People are most likely to be matched accurately with relatives who were about the same age in the photograph as the seeker is now. Some of the photos come with identifying information, others do not, Mr. Patt said.

There are two ways to do the search, he continued. “If you go to the website,, you’ll see a button that says ‘Select and image.’ If you do that, and hit search, you’ll get 10 results that are the most similar to the face in the photo you uploaded. You can get the results of a search of about 34,000 photos — that’s about 170,000 faces — in a couple of seconds. That’s the quick search.

“The full search is accessible by going to the menu. You can sign up and create an account, and then you upload a photo. We run the search over a much larger set of photos — about 500,000 — and you get the results in about a day.

“It’s free to users,” he added.

Mr. Patt and Mr. Limmer funded the site and the technology behind it themselves, they said; they’re working with FJC, “an organization that incubates innovative philanthropies,” Mr. Limmer said. “We were a perfect match for them. They handle the infrastructure.” Numbers to Names has just gone public, and the number of searches has begun to rise dramatically. “Right now, we’re throwing a lot against the wall, to see what sticks,” he said. “We have to fundraise. There is a cost to what we are doing, and right now it is completely self-funded.”

The two men feel that they don’t have much time to spare. “There is an urgency to it,” Mr. Patt said. “My grandmother” — who is very private and doesn’t want her name or any biographical details made public — “was 9 at the beginning of the war.” If any photographs of any of her relatives are to surface, it would be best should that happen soon.

“We also want to partner with schools and universities,” he continued. “The search experience is something that is unforgettable.” If you can match a photo with a name and a place and a family, “you are potentially helping not only the descendant, and maybe even the survivor him or herself, but if there is nobody still alive, you are finding someone who had a life.”

Although the project is brand-new, there is one fairly well-known story its backers already can tell.The rocker Geddy Lee, from the band Rush, is the son of a Holocaust survivor, Mary Weinrib, who died last summer at 95. Numbers to Names was able to identity a photograph of Ms. Weinrib in a DP camp after she was liberated from Bergen-Belsen. Then, the technology helped Mr. Lee find more photos of his extended family in Yad Vashem’s collection.

read more: