Feed a system enough data, algorithms based on specific patterns and features can recognize criteria against a new data set. While it doesn’t replace the investigator at the keyboard, responsible application of artificial intelligence (AI) technology to recognize patterns at scale can be a time saver. I briefly explored optical character recognition (OCR) using the Tesseract package in X-Ways Forensics a few months ago. Combining Tesseract’s OCR technology with XWF’s search capability makes for a very efficient keyword search from files that would ordinarily not be available.
Did you know? According to Google Trends, “DALL-E” reached “peak popularity” between 2022-06-12 and 2022-06-18. DALL-E is a machine learning model that generates images based on text descriptions. Testing DALL-E’s capabilities, I was reminded of a few forensic suites capable of picture categorization using AI.
Magnet Forensics introduced picture categorization with AXIOM v1.1 and added the weapons category with v2.0. Other categorizes included may be viewed in the Magnet AXIOM User Guide.
Belkasoft X is capable of picture classification to detect faces, skin, pornography, guns and text.
X-Ways Software introduced picture categorization with X-Ways Forensics v20.5 using Excire Photo AI. Excire, developed by Pattern Recognition Company GmbH, is a module included with X-Ways Forensics (v20.7, as of 11/15/2022) and supports the following formats: JPEG, PNG, Bitmap, TIFF, non-animated WEBP, GIF, and HEIC (since v20.6). The “weapon” category was added to Excire’s complete list of detectable objects with the release of v20.6 SR-1.
With personal access to Magnet Forensics (6.6.0.33061 – Trial) and X-Ways Forensics (v20.6 SR-4), I explored picture categorization by applying it to a data set, and share thoughts on using combinations of available objects detected for each respective program to focus on photos of potential interest.
Exercise
I was interested in leveraging categories together by filtering for objects identified in a photo that may help with context. XWF already applies this technique with Excire to assist with child abuse investigations. Magnet.AI has a dedicated category that purpose.
I wanted to know if there was value in combining firearms detected in photos with other detectable categories. For example, could detection of a firearm in a photo of an identifiable room belonging to a prohibited person be of interest? AXIOM has a bedroom category. Excire has bedroom, kitchen, hallway and living room categories available. Environmental context may be interesting, too. Generally, a photo of a firearm alone or in an outdoor setting may be benign compared to a photo of a firearm in a city environment in the United States. I also wanted an excuse to combine firearms and digital forensics in a blog post.
A couple of questions:
1. Could AI detect a firearm and sufficiently categorize other detected objects in a photo to determine if it is in an outdoor/nature or urban environment?
2. What other category combinations with firearms might be interesting?
With AXIOM, I considered filtering tags for “Possible Buildings (exterior)” AND “Possible Weapons” to potentially detect firearms in an anomalous environment. Similarly with XWF and Excire, I considered combining categories like “Architecture” OR “City” AND “Weapon”.
Collecting Images for the Data Set
Over the years, I’ve had opportunities to take photographs of firearms. I selected a handful of those photos for this experiment – 34 images.
I do not have pictures of firearms in an urban environment. Where might I source photos of firearms in various environments? Movies and TV entertainment.
You’ve heard of the Internet Movie Database (IMDb), right? Well, the Internet Movie Firearms Database (IMFDb) is a searchable repository of firearms in movies. To supplement my photographs, I selected a handful of samples from IMFDb primarily focused on the perceived environment. A total of the 38 images.
I also included other photos and videos to assess how they may be categorized. I collected 130 files for the initial data set. I added 5 more to test another Excire feature.
Processing the Data Set
Picture Categorization with Magnet AXIOM and Magnet.AI
Enabling picture categorization with AXIOM is simple as checking a few boxes. A handful of categories do require additional processing time. I selected “bedrooms”, “buildings”, “human faces”, “weapons”, “human hands”, “militants”, “vehicles”, and “tattoos”. For a quick introduction, Jamie McQuaid of Magnet Forensics created a video working with Magnet.AI.
Activating Picture Categorization with X-Ways Forensics and Excire
Picture categorization in XWF using Excire may be activated by refining the Volume Snapshot (Specialist | Refine Volume Snapshot | Picture analysis and processing | more options […]). Check the box to Analyze pictures with Excire PhotoAI.
Extract Stills From Video for Picture Analysis in XWF
While AXIOM does capture stills from video and is capable of detecting skin tone from those stills, it did not appear to use Magnet.AI for picture categorization on those stills. XWF does have the ability to extract stills from video and categorize objects from those stills with Excire.
To save time, it may not be appropriate to activate the option to Capture sporadic still images from video with the initial refinement of the volume snapshot for an entire image. Instead, consider activating this option on a specific video (or several) discovered during an examination.
Of the 4 videos I used for this exercise, 3 of the 4 contained a firearm.
To Capture sporadic still images from videos of specific files:
1) Tag the files (videos) of interest.
2) Go to Specialist | Refine Volume Snapshot and check the box for Capture sporadic still images from video and select the appropriate options.
3) Check the box for Picture analysis and processing and select the appropriate options.
4) Check the option to Apply to tagged files only.
It is expected still images are generated as child-objects for the respective files. Each child-object will also have the results of the picture analysis in the column based on the option selected.
Of note: The ability to initiate the RVS process from the context menu to perform Picture analysis and processing with Excire is not available in v20.6 SR-4. Skin tone and b&w detection in pictures work, as expected, however.
Similar to extracting stills from videos, you may also consider selecting the RVS option to Uncover embedded data in various file types to extract the .jpg from files not compatible with Excire, e.g., .orf.
Observations from the Processed Data Set
Of the 38 images from movies or television series that contains a firearm collected from IMFDb.com, 13 (34%) were appropriately tagged as “Possible weapons” (5) or “Possible Militants” (8) in AXIOM, and 18 (47%) were appropriately categorized as a “Weapon” in XWF.
Of my photos (34) that contains a firearm, 28 (82%) were appropriately tagged as “Possible weapons” (26) or “Possible Militants” (2) in AXIOM, and 30 (88%) were appropriately categorized as a “Weapon” in X-Ways Forensics. AXIOM had one false positive for “Possible militants”, and XWF had one false positive for “Weapon”.
AXIOM and XWF did not categorize the firearm contained in a majority of the IMFDb samples I selected. In cinematography, the subject is not always the firearm and filmed at various angles. In contrast, the higher detection rate in my pictures may be attributed to the composition in which the firearm is more likely to be the subject.
I also found it interesting that AXIOM tagged certain photos as “Possible militants” when it was expected to detect “Possible Weapons”.
This significantly reduced my sample and opportunity to observe if AI could detect a firearm and sufficiently categorize other detected objects in a photo to determine if it is in an outdoor/nature or urban environment.
| Firearm Detected – IMFDb | Firearm Detected – mreerie Photos | |
|---|---|---|
| AXIOM | 34% | 82% |
| XWF | 47% | 88% |
City – AXIOM
In AXIOM, I selected categories detected in the case from the IMFDb sample with a logical OR that I perceived to be associated with an urban environment: “Possible buildings (exterior)”, and “Possible Vehicles (cars/trucks/vans/buses)”. This resulted in 8 pictures of 10 possible.
“Possible weapons” were not detected in any of the 8 categorized as “Possible buildings (exterior)” or “Possible Vehicles (cars/trucks/vans/buses)” (0%). However, “Possible militants” were detected in 4 of the 8, which did match the intended target. Overall, a firearm was detected in 4 of the 10 possible pictures I perceived to be associated with an urban environment (40%).
City – XWF
In XWF, I selected categories detected in the case from the IMFDb sample with a logical OR that I perceived to be associated with an urban environment: “Airplane”, “Ambulance”, “Architecture”, “Bridge”, “Building”, “Car”, “City”, “City Lights”, “City Street”, “Concert”, “Golden Gate Bridge”, “Pier”, “Skyscraper”, “Technology”, “Tower”, “Tower Bridge”, “Wall”, “Vehicle”, and “Window”. This resulted in 8 pictures of 10 possible.
A “Weapon” was detected in only 3 of the 8 from that selection – only 1 of which matched the intended target – a still from Heat with a Colt 733. Overall, a firearm was detected in 1 of the 10 possible pictures I perceived to be associated with an urban environment (10%).
Outdoor/Nature – AXIOM
AXIOM does not detect objects that are perceived to be associated with outdoors/nature (NA).
Outdoor/Nature – XWF
In XWF, I selected categories detected in the case from the IMFDb sample with a logical OR that I perceived to be associated with an outdoor/nature environment: “Abandoned Place”, “Agriculture”, “Cabin”, “Clouds”, “Garden”, “Leaf”, “Mountain”, “Nature”, “Ocean”, “Plant”, “Sand”, “Sky”, “Sunset”, “Tree”, “Water”, “Waterfall”, “Waters”, and “Wood”. After excluding 1 false positive, this resulted in 4 of 11 possible pictures I perceived to be associated with an outdoor/nature environment.
A “Weapon” was detected in only 3 from that selection. Overall, a firearm was detected in 3 of the 11 possible pictures I perceived to be associated with an outdoor/nature environment (27%).
| City | Outdoor/Nature | |
|---|---|---|
| AXIOM – “Possible Weapon” | 0% | NA |
| AXIOM – “Possible Militants” | 40% | NA |
| XWF – “Weapon” | 12.5% | 27% |
Based on this potentially flawed sample, the utility of combining categories to search for firearms in an outdoor/nature environment or in an urban environment appears limited.
Lets take look at a potential use for combining other available detected categories – weapons and faces.
Combining Tags Using Filters
AXIOM
AXIOM combines tags using a logical OR; however, it is not capable using a logical AND. To get around this, I create an Excel report of tagged items and filter the list using Excel. My target are the file names with the tags “Possible Weapons” and “Possible Human Faces”, so I filter for rows that only contain “weapons” AND “faces” in the Tags column. This resulted in 4 matches of the 36 possible images from the entire sample (11%).
XWF with Excire
XWF supports logical AND and OR combinations. Since I saved the output in the Report table associations column, I can filter for “Weapon” AND “Face”. After excluding stills extracted from video, this resulted in 11 matches of the 36 possible images from the entire sample (31%).
| Weapons and Faces | |
|---|---|
| AXIOM | 11% |
| XWF | 31% |
Categories “Similar Face to…” AND “Weapon”
Excire also allows you to find faces in images based on samples provided. Matches may be combined with other categories – like “Weapons”. I selected 4 additional samples from IMFDb.org of images of Clint Eastwood that contains a firearm. I used a headshot of Eastwood, by Greg Gorman, for the face to detect.
| 1 | Dirty Harry – Smith & Wesson 29 |
| 2 | The Enforcer – Smith & Wesson 29 |
| 3 | The Outlaw Josey Wales – Colt Walker 1847 |
| 4 | Greg Gorman (1993) |
Identifying Similar Faces with XWF and Excire:
1) Go to Specialist | Refine Volume Snapshot and check the box for Picture analysis and processing and click on more options […].
2) Check the box to Analyze pictures with Excire PhotoAI
3) Check the box to Find particular faces.
4) Select the directory that contains image files of the faces of interest.
5) Adjust the strictness level for matching.
6) Click OK.
7) Verify RVS options, then click OK.
8) A new window will pop up that will display the image files containing the face(s) of interest. Click and drag the mouse on photo to create a square or rectangle frame around the face of interest. Use the PgDn/PgUP keys to navigate through multiple photos. Close the window when complete.
Note: Initial use of this feature will open up a helper text file. If you need to refer to it later, it is located in the XWF directory at \Excire\marker-help.
The headshot of Eastwood matched all 4 photos. In 3 of the 4 images, a weapon was detected. Luck had nothing to do with it, but the angle of the firearm present in 1 of 2 Dirty Harry images is likely the reason it was not detected.
Examples of Other Objects Detected with XWF and Excire
Summary
I sought to explore the picture categorization feature in AXIOM with Magnet.AI, and XWF with Excire. I considered potential use cases that involve combining detected categories to narrow a search using additional context. To evaluate this, I collected images from various sources focused primarily on firearms in outdoor/nature and city environments.
I shared steps to activate the picture categorization feature in AXIOM and XWF, filter for relevant categories, and detect faces with Excire.
My initial testing resulted in a reduced data set. My photographs in which a firearm is the primary subject were more likely to be detected than the photos I selected from the IMFDb. I suspect the composition of the subject, e.g., firearm, is a significant factor of any technology to detect it, which likely contributed to the reduction of the sample size.
While limits are to be expected, combination of tags/categories detected through AI may be a useful technique to quickly identify low-hanging fruit as a basis for further examination/investigation. I had fun with it, anyway.
Edits:
2022-12-02: Excire is functionality is included with XWF v20.7 as of 11/15/2022.
Removed a video link.
2023-01-18: Updated Excire detectable objects link to X-Ways.net
mr. eerie 
One thought on “Exploring AI Assisted Picture Categorization with Magnet Forensics AXIOM and X-Ways Forensics with Excire, Re: Weapons”