Exploring AI Assisted Picture Categorization with Magnet Forensics AXIOM and X-Ways Forensics with Excire, Re: Weapons

Feed a system enough data, algorithms based on specific patterns and features can recognize criteria against a new data set. While it doesn’t replace the investigator at the keyboard, responsible application of artificial intelligence (AI) technology to recognize patterns at scale can be a time saver. I briefly explored optical character recognition (OCR) using the Tesseract package in X-Ways Forensics a few months ago. Combining Tesseract’s OCR technology with XWF’s search capability makes for a very efficient keyword search from files that would ordinarily not be available.

Did you know? According to Google Trends, “DALL-E” reached “peak popularity” between 2022-06-12 and 2022-06-18. DALL-E is a machine learning model that generates images based on text descriptions. Testing DALL-E’s capabilities, I was reminded of a few forensic suites capable of picture categorization using AI.

From OpenAI.com.

Magnet Forensics introduced picture categorization with AXIOM v1.1 and added the weapons category with v2.0. Other categorizes included may be viewed in the Magnet AXIOM User Guide.

Belkasoft X is capable of picture classification to detect faces, skin, pornography, guns and text.

X-Ways Software introduced picture categorization with X-Ways Forensics v20.5 using Excire Photo AI. Excire, developed by Pattern Recognition Company GmbH, is a module included with X-Ways Forensics (v20.7, as of 11/15/2022) and supports the following formats: JPEG, PNG, Bitmap, TIFF, non-animated WEBP, GIF, and HEIC (since v20.6). The “weapon” category was added to Excire’s complete list of detectable objects with the release of v20.6 SR-1.

With personal access to Magnet Forensics (6.6.0.33061 – Trial) and X-Ways Forensics (v20.6 SR-4), I explored picture categorization by applying it to a data set, and share thoughts on using combinations of available objects detected for each respective program to focus on photos of potential interest.

Exercise

I was interested in leveraging categories together by filtering for objects identified in a photo that may help with context. XWF already applies this technique with Excire to assist with child abuse investigations. Magnet.AI has a dedicated category that purpose.

I wanted to know if there was value in combining firearms detected in photos with other detectable categories. For example, could detection of a firearm in a photo of an identifiable room belonging to a prohibited person be of interest? AXIOM has a bedroom category. Excire has bedroom, kitchen, hallway and living room categories available. Environmental context may be interesting, too. Generally, a photo of a firearm alone or in an outdoor setting may be benign compared to a photo of a firearm in a city environment in the United States. I also wanted an excuse to combine firearms and digital forensics in a blog post.

A couple of questions:
1. Could AI detect a firearm and sufficiently categorize other detected objects in a photo to determine if it is in an outdoor/nature or urban environment?
2. What other category combinations with firearms might be interesting?

With AXIOM, I considered filtering tags for “Possible Buildings (exterior)” AND “Possible Weapons” to potentially detect firearms in an anomalous environment. Similarly with XWF and Excire, I considered combining categories like “Architecture” OR “City” AND “Weapon”.

Collecting Images for the Data Set

Over the years, I’ve had opportunities to take photographs of firearms. I selected a handful of those photos for this experiment – 34 images.

I do not have pictures of firearms in an urban environment. Where might I source photos of firearms in various environments? Movies and TV entertainment.

You’ve heard of the Internet Movie Database (IMDb), right? Well, the Internet Movie Firearms Database (IMFDb) is a searchable repository of firearms in movies. To supplement my photographs, I selected a handful of samples from IMFDb primarily focused on the perceived environment. A total of the 38 images.

Intended Environment1234567891011
IndoorDie Hard – PPKEagleheart – GlockThe Room – Beretta 92 FS InoxThe Office (USA) – Remington Model 1875The Punisher – Mossberg 590MThe Titanic – 1911Law & Order – MultipleI Am Legend (2007) – Beretta 92 FS I Am Legend (2007) – Beretta 92 FSFringe – GlockMr. & Mrs. Smith – P7M13
More IndoorChuck – Suction Cup PistolChuck – Glock 21Chuck – Mossbuerg 590A1Chuck – Beretta 92 FSChuck – Gun Display24 (Japan) – Beretta 92 FS<blank><blank><blank><blank><blank>
Outdoor/NatureTropic Thunder – Ithaca 37American Sniper – Remington Model 700Training Day – Smith & Wesson M66No Country for Old Men – Remington Model 700Wind River – Marlin 1895SBLThe Deer Hunter – Remington 700 BDLThe Office – Remington M70 Southern Comfort – Remington Rolling BlockThose Whole Kill – Daisy Red RiderTrue Lies – Glock 19Rambo FB2 – M60E2
Outdoor | CityHeat – Colt 654Heat– Colt 654Heat – Colt 733Heat – Colt 733Heat – FNCCaptain America: TWS – Colt M4A1Last Action Hero – Desert EagleLast Action Hero – SIG Sauer P230Police Academy – Smith & Wesson M13Southland – HK USP <blank>
Selected images from https://www.imfdb.org/

I also included other photos and videos to assess how they may be categorized. I collected 130 files for the initial data set. I added 5 more to test another Excire feature.

Processing the Data Set

Picture Categorization with Magnet AXIOM and Magnet.AI

Enabling picture categorization with AXIOM is simple as checking a few boxes. A handful of categories do require additional processing time. I selected “bedrooms”, “buildings”, “human faces”, “weapons”, “human hands”, “militants”, “vehicles”, and “tattoos”. For a quick introduction, Jamie McQuaid of Magnet Forensics created a video working with Magnet.AI.

As shown with AXIOM Process 6.6.0.33061.
Post processing with AXIOM Examine 6.6.0.33.061.

Activating Picture Categorization with X-Ways Forensics and Excire

Picture categorization in XWF using Excire may be activated by refining the Volume Snapshot (Specialist | Refine Volume Snapshot | Picture analysis and processing | more options []). Check the box to Analyze pictures with Excire PhotoAI.

RVS
Picture Analysis options for Excire.
The ability to categorize categories as irrelevant or notable exists. Some combinations of known bad categories are predefined.
Number of matches detected for each respective category after the RVS process is completed.

Extract Stills From Video for Picture Analysis in XWF

While AXIOM does capture stills from video and is capable of detecting skin tone from those stills, it did not appear to use Magnet.AI for picture categorization on those stills. XWF does have the ability to extract stills from video and categorize objects from those stills with Excire.

To save time, it may not be appropriate to activate the option to Capture sporadic still images from video with the initial refinement of the volume snapshot for an entire image. Instead, consider activating this option on a specific video (or several) discovered during an examination.

Of the 4 videos I used for this exercise, 3 of the 4 contained a firearm.

To Capture sporadic still images from videos of specific files:
1) Tag the files (videos) of interest.
2) Go to Specialist | Refine Volume Snapshot and check the box for Capture sporadic still images from video and select the appropriate options.
3) Check the box for Picture analysis and processing and select the appropriate options.
4) Check the option to Apply to tagged files only.

It is expected still images are generated as child-objects for the respective files. Each child-object will also have the results of the picture analysis in the column based on the option selected.

15 stills were generated as child-objects for the video IMG_20200215-095239.MOV.
Results of Picture analysis and processing are shown as report table associations.

Of note: The ability to initiate the RVS process from the context menu to perform Picture analysis and processing with Excire is not available in v20.6 SR-4. Skin tone and b&w detection in pictures work, as expected, however.

RVS from the context menu.
Excire options are absent from Picture analysis and processing options if the RVS is summoned from context menu.

Similar to extracting stills from videos, you may also consider selecting the RVS option to Uncover embedded data in various file types to extract the .jpg from files not compatible with Excire, e.g., .orf.

.orf is an available file type to uncover embedded data from, by default. The thumbnail uncovered from a .psd file I tested is 160x106px, which is under the size Excire will analyze by default.
Two .jpg files were uncovered, but only one file was larger than 160×160 px to be analyzed with Excire. From Kowloon Public Pier, Hong Kong, China.

Observations from the Processed Data Set

Of the 38 images from movies or television series that contains a firearm collected from IMFDb.com, 13 (34%) were appropriately tagged as “Possible weapons” (5) or “Possible Militants” (8) in AXIOM, and 18 (47%) were appropriately categorized as a “Weapon” in XWF.

Of my photos (34) that contains a firearm, 28 (82%) were appropriately tagged as “Possible weapons” (26) or “Possible Militants” (2) in AXIOM, and 30 (88%) were appropriately categorized as a “Weapon” in X-Ways Forensics. AXIOM had one false positive for “Possible militants”, and XWF had one false positive for “Weapon”.

A false positive for “Possible militants” in AXIOM. Walking around in Huilai, Guangdong, China.
A false positive for “Weapon” in XWF. illy CaffĂ© at The Venetian Resort, Las Vegas, Nevada, USA.

AXIOM and XWF did not categorize the firearm contained in a majority of the IMFDb samples I selected. In cinematography, the subject is not always the firearm and filmed at various angles. In contrast, the higher detection rate in my pictures may be attributed to the composition in which the firearm is more likely to be the subject.

I also found it interesting that AXIOM tagged certain photos as “Possible militants” when it was expected to detect “Possible Weapons”.

Thumbnail view in AXIOM of pictures tagged with “Possible weapons”. The stars marks samples from the IMFDb.
Gallery mode in XWF. More samples from the IMFDb. The x marks samples AXIOM and XWF did not detect a firearm. The favicon of Magnet Forensics and X-Ways Software identify the respective suite that identified a firearm of that picture.

This significantly reduced my sample and opportunity to observe if AI could detect a firearm and sufficiently categorize other detected objects in a photo to determine if it is in an outdoor/nature or urban environment.

Firearm Detected – IMFDbFirearm Detected – mreerie Photos
AXIOM 34%82%
XWF47%88%

City – AXIOM

In AXIOM, I selected categories detected in the case from the IMFDb sample with a logical OR that I perceived to be associated with an urban environment: “Possible buildings (exterior)”, and “Possible Vehicles (cars/trucks/vans/buses)”. This resulted in 8 pictures of 10 possible.

“Possible weapons” were not detected in any of the 8 categorized as “Possible buildings (exterior)” or “Possible Vehicles (cars/trucks/vans/buses)” (0%). However, “Possible militants” were detected in 4 of the 8, which did match the intended target. Overall, a firearm was detected in 4 of the 10 possible pictures I perceived to be associated with an urban environment (40%).

A green check are matches for “Possible militants” AND ” Possible weapons” in AXIOM.

City – XWF

In XWF, I selected categories detected in the case from the IMFDb sample with a logical OR that I perceived to be associated with an urban environment: “Airplane”, “Ambulance”, “Architecture”, “Bridge”, “Building”, “Car”, “City”, “City Lights”, “City Street”, “Concert”, “Golden Gate Bridge”, “Pier”, “Skyscraper”, “Technology”, “Tower”, “Tower Bridge”, “Wall”, “Vehicle”, and “Window”. This resulted in 8 pictures of 10 possible.

A “Weapon” was detected in only 3 of the 8 from that selection – only 1 of which matched the intended target – a still from Heat with a Colt 733. Overall, a firearm was detected in 1 of the 10 possible pictures I perceived to be associated with an urban environment (10%).

Outdoor/Nature – AXIOM

AXIOM does not detect objects that are perceived to be associated with outdoors/nature (NA).

Outdoor/Nature – XWF

In XWF, I selected categories detected in the case from the IMFDb sample with a logical OR that I perceived to be associated with an outdoor/nature environment: “Abandoned Place”, “Agriculture”, “Cabin”, “Clouds”, “Garden”, “Leaf”, “Mountain”, “Nature”, “Ocean”, “Plant”, “Sand”, “Sky”, “Sunset”, “Tree”, “Water”, “Waterfall”, “Waters”, and “Wood”. After excluding 1 false positive, this resulted in 4 of 11 possible pictures I perceived to be associated with an outdoor/nature environment.

A “Weapon” was detected in only 3 from that selection. Overall, a firearm was detected in 3 of the 11 possible pictures I perceived to be associated with an outdoor/nature environment (27%).

CityOutdoor/Nature
AXIOM – “Possible Weapon”0% NA
AXIOM – “Possible Militants”40%NA
XWF – “Weapon”12.5% 27%

Based on this potentially flawed sample, the utility of combining categories to search for firearms in an outdoor/nature environment or in an urban environment appears limited.

Lets take look at a potential use for combining other available detected categories – weapons and faces.

Combining Tags Using Filters

AXIOM

AXIOM combines tags using a logical OR; however, it is not capable using a logical AND. To get around this, I create an Excel report of tagged items and filter the list using Excel. My target are the file names with the tags “Possible Weapons” and “Possible Human Faces”, so I filter for rows that only contain “weapons” AND “faces” in the Tags column. This resulted in 4 matches of the 36 possible images from the entire sample (11%).

Filtered for “Possible Weapons” OR “Possible Human Faces”
Creating a report of the File Name and Tags for pictures
Filter for tags in Excel.

XWF with Excire

XWF supports logical AND and OR combinations. Since I saved the output in the Report table associations column, I can filter for “Weapon” AND “Face”. After excluding stills extracted from video, this resulted in 11 matches of the 36 possible images from the entire sample (31%).

After filtering out extracted stills from videos.
Weapons and Faces
AXIOM11%
XWF31%

Categories “Similar Face to…” AND “Weapon”

Excire also allows you to find faces in images based on samples provided. Matches may be combined with other categories – like “Weapons”. I selected 4 additional samples from IMFDb.org of images of Clint Eastwood that contains a firearm. I used a headshot of Eastwood, by Greg Gorman, for the face to detect.

1Dirty Harry Smith & Wesson 29
2The Enforcer – Smith & Wesson 29
3The Outlaw Josey Wales – Colt Walker 1847
4Greg Gorman (1993)
More samples from the IMFDb.org.

Identifying Similar Faces with XWF and Excire:
1) Go to Specialist | Refine Volume Snapshot and check the box for Picture analysis and processing and click on more options […].
2) Check the box to Analyze pictures with Excire PhotoAI
3) Check the box to Find particular faces.
4) Select the directory that contains image files of the faces of interest.
5) Adjust the strictness level for matching.
6) Click OK.
7) Verify RVS options, then click OK.
8) A new window will pop up that will display the image files containing the face(s) of interest. Click and drag the mouse on photo to create a square or rectangle frame around the face of interest. Use the PgDn/PgUP keys to navigate through multiple photos. Close the window when complete.

Note: Initial use of this feature will open up a helper text file. If you need to refer to it later, it is located in the XWF directory at \Excire\marker-help.

The headshot of Eastwood matched all 4 photos. In 3 of the 4 images, a weapon was detected. Luck had nothing to do with it, but the angle of the firearm present in 1 of 2 Dirty Harry images is likely the reason it was not detected.

Combining “Weapon” AND “Similar Face to” would match 3 of the 4 possible in this small sample.

Examples of Other Objects Detected with XWF and Excire

“Cause every tool is a weapon if you hold it right.” – Ani diFranco

Summary

I sought to explore the picture categorization feature in AXIOM with Magnet.AI, and XWF with Excire. I considered potential use cases that involve combining detected categories to narrow a search using additional context. To evaluate this, I collected images from various sources focused primarily on firearms in outdoor/nature and city environments.

I shared steps to activate the picture categorization feature in AXIOM and XWF, filter for relevant categories, and detect faces with Excire.

My initial testing resulted in a reduced data set. My photographs in which a firearm is the primary subject were more likely to be detected than the photos I selected from the IMFDb. I suspect the composition of the subject, e.g., firearm, is a significant factor of any technology to detect it, which likely contributed to the reduction of the sample size.

While limits are to be expected, combination of tags/categories detected through AI may be a useful technique to quickly identify low-hanging fruit as a basis for further examination/investigation. I had fun with it, anyway.

Edits:

12/02/2022: Excire is functionality is included with XWF v20.7 as of 11/15/2022.
Removed a video link.

Advertisement

One thought on “Exploring AI Assisted Picture Categorization with Magnet Forensics AXIOM and X-Ways Forensics with Excire, Re: Weapons

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.