Accessing Windows Live Photo Gallery Face API

While working on a tool to automatically organize media files (move new images to their correct directory) i though: could i use the Windows Live Photo Gallery (WLPG) face recognition engine to my advantage? The photos i want to organize are people, and i usually try to mark the faces with WLPG before running my program (which uses the faces embeded in the jpeg along other metadata to help select the final directory). Maybe i could use the face engine to recognize some faces inside my program.
As it happens, the API is not well prepared to be used by others, and the recognition engine is not perfect, but i managed to use it to some level. This is how i did it using Embarcadero RAD Studio 2010 (Delphi), and with Windows Live Photo Gallery 2011.
First steps
First, you have to import the Type Library “WLPG FaceRecognitionObjects” (which is in WLXFaceRecognition.dll on the WLPG directory). This will give you access to the exposed API.
To process an image file (for detection or recognition) you have to load it into an IWICBitmap object to access its data. I haven’t used exception check code to make it more clear, but you should capture exceptions and make sure you free any objects you create.
var theImageManager: IImageManager; theFaceWICBitmap: FaceRecognitionObjects_TLB.IWICBitmap; theImageData: IImageData; theWICImage: Graphics.TWICImage; begin theWICImage := TWICImage.Create; theWICImage.LoadFromFile(filename); theFaceWICBitmap := FaceRecognitionObjects_TLB.IWICBitmap(theWICImage.Handle); theImageManager := CoImageManager.Create; theImageManager.CreateImageDataFromWICBitmap(theFaceWICBitmap, theImageData);
The theImageData variable will be used on the following calls to detect or recognize faces.
Face Detection
Detection is an easy task.
var theFaceDetection: IFaceDetection; theFaceRegionSet: IFaceRegionSet; theFaceRegion: IFaceRegion; faceCount: Cardinal; theFaceRegionRectangle: FaceRegionRect; theFacePose: FacePose; f: Integer; begin theFaceDetection := CoFaceDetection.Create; if theFaceDetection.RunFaceDetection (theImageData, theFaceRegionSet) = S_OK then begin if theFaceRegionSet <> nil then theFaceRegionSet.GetCount(faceCount); for f := 0 to Integer(faceCount) - 1 do if theFaceRegionSet.GetAt(f, theFaceRegion) = S_OK then begin if theFaceRegion.GetFaceRegionRect(theFaceRegionRectangle) = S_OK then begin // theFaceRegionRectangle contains the face rectangle coordinates (left, top, width, height) in percentages end; if theFaceRegion.GetFacePose(theFacePose) = S_OK then begin // theFacePose contains the detected face pose (see the corresponding enumerations in the type library) end; end; end; end;
For each face you can use the GetFaceRegionRect and GetFacePose methods to get the rectangle and the detected pose (angle of the face). The face objects have other methods but since we only asked to detect the faces they will not return useful data at this point. Unfortunately just detecting a face is not that useful if you cannot recognize it.
Face Recognition
Face recognition is a very different beast to deal with. Lets see first the code.
type TFaceRecommendations = array [0 .. 0] of FaceRecommendation; var theFaceRecognitionPipeline: IFaceRecognitionPipeline; theFaceRegionSet: IFaceRegionSet; theFaceRegion: IFaceRegion; faceCount: Cardinal; theFaceRegionRectangle: FaceRegionRect; theFacePose: FacePose; theFacePersonId: Integer; theFacePersonName: WideString; pFaceRecommendations: ^TFaceRecommendations; faceRecommendationsCount: Cardinal; f, r: Integer; begin theFaceRecognitionPipeline := CoFaceRecognitionPipeline.Create; theFaceRecognitionPipeline.SetMatchThreshold(MatchThresholdDefault); if theFaceRecognitionPipeline.RecognizeFacesFromImage (theImageData, theFaceCache, theFaceRegionSet) = S_OK then begin if theFaceRegionSet <> nil then theFaceRegionSet.GetCount(faceCount); for f := 0 to Integer(faceCount) - 1 do if theFaceRegionSet.GetAt(f, theFaceRegion) = S_OK then begin if theFaceRegion.GetFaceRegionRect(theFaceRegionRectangle) = S_OK then begin // theFaceRegionRectangle contains the face rectangle coordinates (left, top, width, height) in percentages end; if theFaceRegion.GetFacePose(theFacePose) = S_OK then begin // theFacePose contains the detected face pose (see the corresponding enumerations in the type library) end; theFacePersonId := 0; if faceregion.GetPersonId(theFacePersonId) <> S_OK then begin // theFacePersonId contains the internal id of the detected person end; theFacePersonName := ''; if faceregion.GetPersonName(theFacePersonName) <> S_OK then begin // theFacePersonName contains the name of the detected person end; // Next we read the person recommendations with their confidence level pFaceRecommendations := nil; // the method will allocate memory if theFaceRegion.GetRecommendations (PUserType3(pFaceRecommendations), faceRecommendationsCount) = S_OK then if faceRecommendationsCount > 0 then begin for r := 0 to Integer(faceRecommendationsCount) - 1 do if pFaceRecommendations^[r].nPersonId <> 0 then // a personId of 0 means no one. begin // you get the confidence level from pFaceRecommendations^[r].flConfidence (a percentage). end; end; end; end; end;
It looks very similar to the detection code, but if you look closely you will notice that it only returns a numeric Id for the recognized person(s), and the main method has an additional parameter (theFaceCache). The GetPersonName method only returns something when the engine is 100% sure of the recognition. So, where do we get the person recommendations names? and what is that face cache?
Face exemplar cache
That face cache is the object that holds the faces information, allows the engine to recognize, and maps a person Id with its name. And to properly use the recognized faces from WLPG we will need to populate it from the application database, since the API does not give us access to that data.
We will have to define a class based on the IExemplarCache interface that will hold the face information from the database.
type TPersonFaceData = class public PersonId: cardinal; Representation: TBytes; FaceRep:array of FaceRepresentation; constructor Create(const id: integer; rep: TBytes); procedure AddRepresentation (rep:TBytes); destructor Destroy; override; end; TExemplarCache = class(TInterfacedObject, IExemplarCache) private cache: TObjectList; personnames: TStringList; statusCacheStamp:integer; public function GetPersonCount(out puPersonCount: LongWord): HResult; stdcall; function GetPersonData(uPersonIndex: LongWord; out pnPersonId: integer; out ppFaceRepresentations: PUserType4; out puRepresentationCount: LongWord) : HResult; stdcall; function GetCacheStamp(out pdwCacheStamp: LongWord): HResult; stdcall; constructor Create; destructor Destroy; override; procedure ImportFromWLPG; function GetPersonName(uPersonId: integer): string; end; { TPersonFaceData } procedure TPersonFaceData.AddRepresentation(rep: TBytes); begin setlength (FaceRep, length(FaceRep)+1); FaceRep[high(FaceRep)].uByteCount := length(rep); GetMem (FaceRep[high(FaceRep)].pbBytes, FaceRep[high(FaceRep)].uByteCount); move (rep[0], FaceRep[high(FaceRep)].pbBytes^, FaceRep[high(FaceRep)].uByteCount); end; constructor TPersonFaceData.Create(const id: integer; rep: TBytes); begin PersonId := id; Representation := rep; setlength (FaceRep, 1); FaceRep[0].uByteCount := length(rep); GetMem (FaceRep[0].pbBytes, FaceRep[0].uByteCount); move (rep[0], FaceRep[0].pbBytes^, FaceRep[0].uByteCount); end; destructor TPersonFaceData.Destroy; var I: Integer; begin for I := 0 to high(FaceRep) do FreeMem (FaceRep[I].pbBytes); inherited; end; { TExemplarCache } constructor TExemplarCache.Create; begin cache := TObjectList.Create; personnames := TStringList.Create; end; destructor TExemplarCache.Destroy; begin personnames.Free; cache.Free; inherited; end; function TExemplarCache.GetCacheStamp(out pdwCacheStamp: LongWord): HResult; begin pdwCacheStamp := statusCacheStamp; Result := S_OK; end; function TExemplarCache.GetPersonCount(out puPersonCount: cardinal): HResult; begin puPersonCount := cache.Count; Result := S_OK; end; function TExemplarCache.GetPersonData(uPersonIndex: cardinal; out pnPersonId: integer; out ppFaceRepresentations: PUserType4; out puRepresentationCount: cardinal): HResult; begin if (uPersonIndex < cache.Count) then begin pnPersonId := cache[uPersonIndex].PersonId; puRepresentationCount:=length(cache[uPersonIndex].FaceRep); ppFaceRepresentations := @(cache[uPersonIndex].FaceRep[0]); Result := S_OK; end else begin pnPersonId := 0; ppFaceRepresentations:=nil; puRepresentationCount:=0; Result := S_FALSE; end; end; function TExemplarCache.GetPersonName(uPersonId: integer): string; var i: Integer; begin if (uPersonId > 0) and (uPersonId-1 < personnames.Count) then begin i:=PersonNames.IndexOfObject(pointer(uPersonId)); if i >= 0 then Result := PersonNames[i] else Result:=''; end else Result := ''; end; procedure TExemplarCache.ImportFromWLPG; var conn: TADOConnection; query: TADOQuery; pName: TField; pId: TField; pData: TField; I: Integer; found: Boolean; wlpg_dir: String; begin cache.Clear; personnames.Clear; try wlpg_dir:=GetSpecialFolderLocation(CSIDL_LOCAL_APPDATA)+'\Microsoft\Windows Live Photo Gallery\'; // read face exemplar data conn:=TADOConnection.Create(nil); query:=TADOQuery.Create(nil); try conn.ConnectionString := 'Provider=Microsoft.SQLLITE.MOBILE.OLEDB.3.0;Data Source="'+wlpg_dir+'FaceExemplars.ed1";SSCE:Max Database Size=4000'; conn.LoginPrompt := false; query.Connection := conn; query.CursorLocation := clUseServer; query.CursorType := ctOpenForwardOnly; query.LockType := ltReadOnly; query.SQL.Text := 'SELECT PersonId, Data FROM tblExemplar'; query.Open; try while not query.Eof do begin try pId := query.FieldByName('PersonId'); pData := query.FieldByName('Data'); if (pId <> nil) and (pData <> nil) then begin found := false; for I := 0 to cache.Count - 1 do if cache[i].PersonId = pId.AsInteger then begin found := true; cache[i].AddRepresentation (pData.AsBytes); break; end; if not found then cache.Add (TPersonFaceData.Create (pId.AsInteger, pData.AsBytes)); end; except end; query.Next; application.processmessages; end; finally query.Close; end; query.SQL.Text := 'SELECT ExemplarCacheStamp FROM tblStatus'; query.Open; try if not query.Eof then statusCacheStamp := query.FieldByName('ExemplarCacheStamp').AsInteger; finally query.Close; end; finally query.Free; conn.Close; conn.Free; end; // read person names conn:=TADOConnection.Create(nil); query:=TADOQuery.Create(nil); try conn.ConnectionString := 'Provider=Microsoft.SQLLITE.MOBILE.OLEDB.3.0;Data Source="'+wlpg_dir+'Pictures.pd6";SSCE:Max Database Size=4000'; conn.LoginPrompt := false; query.Connection := conn; query.CursorType := ctOpenForwardOnly; query.LockType := ltReadOnly; query.CursorLocation := clUseServer; query.SQL.Text := 'SELECT PersonId, Name FROM tblPerson'; query.Open; try while not query.Eof do begin try pName := query.FieldByName('Name'); pId := query.FieldByName('PersonId'); if (pName <> nil) and (pId <> nil) then if pId.AsInteger <> 0 then personnames.AddObject (pName.AsString, pointer(pId.AsInteger)); except end; query.Next; application.processmessages; end; finally query.Close; conn.Close; end; finally query.Free; conn.Free; end; except on e:exception do begin cache.Clear; personnames.Clear; end; end; end;
An object of type TExemplarCache, populated using the ImportFromWLPG method will be passed to the RecognizeFacesFromImage method, which will use its data to recognize the faces. Then, for each face we can use the GetPersonName method of the cache to know the name associated to the id returned.
Unfortunately, the recognition engine is very slow and needs to be properly trained (within WLPG) to return good confidence levels for the faces.
It is recommended to not run WLPG at the same time you call the ImportFromWLPG method. The code could raise an exception because the database is already in use.
UPDATE: My WLPG database got corrupted. Not sure if it was because i ran some tests while having WLPG open. Thankfully, i had a backup. I suggest you make sure WLPG is not running, or make backups of the entire database directory.
I hope Microsoft will open more their API in a next version, to give easier and greater access to the face exemplar cache.