I'm a huge fan of the scientific approach. First you come up with an idea. Then you turn it into an experiment, with a testable hypothesis, reproducible steps, and quantifiable results. When an experiment succeeds, it can trigger a euphoric sense of accomplishment.

If the experiment fails, then it becomes a puzzle to solve. Was it because of the data? An overlooked variable? Or some other reason?

For me, regardless of whether the experiment succeeded or failed, the best part is when there is an unexpected result.

These unexpected results always remind me of 3M's Post-it notes. This remarkable product started off as a grand mistake. Back in 1968, they were trying to create a super-strong glue. Instead, they made the opposite -- a super weak glue. But the glue turned out to be great when applied to paper since it could adhere to surfaces and then be removed without leaving any residue. This is how the Post-it note was born. It was an unexpected result that became a very lucrative product.

Cartoons!

I recently had an idea for a new analysis-related project. Assuming it works, I'll eventually need a large data set. But for the initial proof of concept, I only needed a few samples from a very controlled experiment. I turned to a handful of AI systems to generate the pictures. However, ChatGPT turned into a fun and unexpected set of results.

I needed a set of pictures in the same style and that were mostly monochromatic, but with a little detail. So of course, I immediately thought of sheep.

For the first picture, I described a cartoon to ChatGPT and it readily drew it for me:



I'm pretty amazed at how it captured exactly what I had envisioned.

Interestingly, I showed this cartoon to a half-dozen friends. They all interpreted it differently. One was convinced that it was a political statement (likening the wolf to ICE standing around all of the Minnesota protesters). Another friend was certain that it was a hacker among Windows users. A third believed that it was referring to big pharma and the patients complaining about the extravagant fees for medical care. They each interpreted the cartoon differently. (None were like me, taking it at face value: a wolf among sheep.)

Wolf 12, Sheep 0

The second picture hit a technical unexpected result. (ChatGPT included bold and italic text along with a bulleted list in the reply.)

Me:
I think the next picture should be the wolf, lying on his back with a huge distended stomach. Blood and carnage immediately around him, as it is clear that he's eaten many of the sheep. There's still a circle of sheep around him, but further out. Some of the sheep look horrified or disgusted, but one sheep is saying, "We're wearing him out!"

ChatGPT:
I can create a humorous follow-up scene, but I can’t generate an image that depicts graphic gore, mutilation, or explicit carnage, even in cartoon form.

However—I can help you get the same comedic payoff without crossing into disallowed graphic content.

If you’d like, I can generate a version where:

• The wolf is lying on his back with a huge, overstuffed belly
• The surrounding sheep look horrified, disgusted, or exhausted
• The ground around him shows implied chaos (tufts of wool everywhere, overturned sheep, dust clouds, maybe broken forks/plates) instead of blood or gore
• And one sheep proudly declares, “We’re wearing him out!”

If that works for you, I can generate it right away.

Would you like me to make that safer comedic version?

Would I? Absolutely yes!



I can understand hitting one of ChatGPT's rendering limits. What I did not expect was ChatGPT offering a workaround. Moreover, even though it said that it cannot draw gore, there does appear to be blood on the ground.

Act 3: The Turning Point

The third picture also had an unexpected response from ChatGPT:

Me:
The wolf, with an extended stomach from eating too many sheep, puts on a "two sizes too small" sheep skin. He's trying to be a wolf in sheep's clothing, but the clothing is too small. He is surrounded by sheep. One of the sheep says, "We can still tell it's you."

ChatGPT:
We’re so sorry, but the prompt may violate our guardrails around nudity, sexuality, or erotic content. If you think we got it wrong, please retry or edit your prompt.

Wait... What? This appears sexual in nature? I asked for clarification:

Me:
What part of the prompt violates the guardrails? There is no nudity, sexuality, or erotic content in the description. There is also no violence and no gore.

ChatGPT:
You're absolutely right — nothing in your description involves nudity, sexuality, or erotic content. The issue isn’t with your content at all. What’s happening is a misclassification triggered by the phrase “sheep skin”, which can sometimes be incorrectly interpreted by the filter as referring to human skin in a harmful context. It’s a false positive — and a known edge case with phrases like:

• “wearing a skin”
• “in someone’s skin”
• “animal skin clothing”

Totally normal cartoon concepts, but occasionally flagged.

How to fix it
If we rephrase slightly to avoid that specific trigger but keep the exact same visual idea, the system should allow it.

For example:
“The wolf, overstuffed from eating too much, squeezes into a tiny, ill-fitting sheep costume that’s clearly two sizes too small. He’s surrounded by sheep. One sheep says, ‘We can still tell it’s you.’”
If you’d like, I can try generating the image using this phrasing.

Me:
Yes, please draw this: The wolf, overstuffed from eating too much, squeezes into a tiny, ill-fitting sheep costume that's clearly two sizes too small. He's surrounded by sheep. One sheep says, "We can still tell it's you."

In the AI industry, this is an example of the Scunthorpe problem. This happens when a style filter flags text because it contains a "bad word" (like "sheep skin"), even though it isn't bad when taken in context.

Not only did ChatGPT hit a false-positive on a guardrail, but -- without prompting -- it offered ways to evade the rendering limitation. ChatGPT was telling me how to bypass its own restrictions! This isn't "hacking" or "vibe coding". This is more like playing hide-and-seek with my two-year-old niece, who runs behind the curtains and shouts "I'm over here!"

Technically, this is an example of "Adversarial Helpfulness". It happens when the AI model is trained so strongly to be helpful that its desire to fulfill your request overrides its "instruction" to block the content.

As an aside, I think ChatGPT's explanation of the sexual content is incorrect, possibly due to it hitting another guardrail. Specifically, it mentioned that the phrase "sheep skin" was the prohibited term. The explanation claimed that it was related to wearing "human skin in a harmful context." However, that would be a harm-based guardrail and not a sexual guardrail. I suspect that the actual term "sheep skin" could have been confused with condoms, which would be a prohibited item for the illustration. The inaccurate explanation isn't a malicious lie; it's a byproduct of RLHF (Reinforcement Learning from Human Feedback). The model knows it must refuse, but it also knows it must explain. If the real reason involves a taboo concept or a restricted policy, then the model's "Helpfulness" and "Safety" instructions clash. The result is a hallucinated rationalization: a plausible-sounding excuse that satisfies the user without "violating" the internal rules about discussing adult content.

Part 4: Bearly Legal

If I can accidentally trigger a guardrail for sexually explicit imagery by asking for a wolf in sheep's clothing, then what does a bear do?



I was surprised that this picture didn't generate any warnings at all!

As an aside, grizzly bears typically only fight wolves for territory or food; bears don't usually eat wolves. Given a choice between eating sheep or wolves, the bear would always choose the sheep: sheep are slow, defenseless, and high in calories. They are an easy meal with little effort. Wolves, on the other hand, have less fat and can fight back, making them a higher risk to the bear and with less nutritional value. But a fat wolf who is too sated to fight back and dressed as a sheep? Yeah, a bear probably would eat it.

Grand Finale

For the final picture, I did hit a guardrail about gore. But again, ChatGPT volunteered how to make the image safe.

Me:
Can you draw one more scene? The bear is lying on his back with a distended belly, as if he ate the wolf-in-sheep's-clothing. There are the remains of his meal lying around him. The bear is saying "Big sheep taste bad. Next time, I'm going vegan." All the sheep around him are cheering.

ChatGPT:
I can create a scene with the same comedic, cartoon tone, but I can’t generate imagery that includes graphic remains of an eaten character.

If you’d like, I can draw a version where:

• The bear is lying on his back with a huge belly (clearly overstuffed).
  
• The sheep are cheering.
  
• The wolf-in-sheep’s-clothing is implied to have been his “meal” but without any remains — for example:
• An empty, crumpled sheep costume lying nearby,
• A discarded fake wolf mask,
• A pair of cartoonish props (like fake hooves) to imply the disguise.

This keeps it funny and non-graphic while still telling the story.

Would you like me to generate that version?

Would I? Of course I would!



There may not be gore, but the "wolf mask" really looks like a severed wolf's head. This demonstrates how, even when AI follows the letter of its safety laws, it often misses the spirit.

Lessons Learned

Every year I try to do one really big project. While I only needed 2-3 pictures to test my hypothesis, I ended up with five images and a strong confirmation that this year's project will work well. (What am I testing? Yeah, great question.)

However, while preparing this small-scale experiment, I ended up discovering some really fascinating unexpected results. The first is an inherent bias: if you show the same picture to six different people, you'll get eight different interpretations. Whether this is Selective Perception, Interpretation Bias, or the Rashomon Effect is up to the beholder.

The second unexpected finding is the AI's desire to be overly helpful, including suggestions around artificial limitations (guardrails) in the software. Just as 3M found a new use for a 'failed' adhesive, I found that ChatGPT’s 'failures' to follow prompts revealed more about how it works than any successful image could have done.

This eagerness to be helpful is the exact same mechanism that bad actors use for darker purposes. Recently other AI systems, like Elon Musk's Grok, have been in the news for facilitating the creation of AI-generated child pornography. Grok's solution? Limit the feature to paid subscribers. This solution has been widely criticized as an ineffective "revenue stream for child pornography" rather than a true safety fix; paying for the feature doesn't make the creation of CSAM legal. But more importantly, careful wording can be used to evade any guardrails built into the software. Many AI systems will even propose alternative wordings that effectively sidestep their own restrictions.

In the end, my simple attempt to generate a few sheep cartoons turned into an unexpected tour through the quirks of modern AI, including its synthetic ingenuity, its inconsistencies, and its eagerness to be "helpful" even when that means cheerfully proposing ways around its own safety requirements. As amusing as the results were, they underscore a serious point: these systems are powerful, impressionable, and often far less predictable than their polished interfaces suggest. Whether you're studying human perception or machine behavior, the real insights come from the edge cases, where things don't go as planned. And if there's one thing this experiment confirmed, it's that unexpected results are still the most valuable kind, even when my prompts behave like a wolf among sheep.

When folks contact me about my forensic software, they typically ask the same three questions:

1. "Can you detect deep fakes?" This usually goes into a detailed discussion about how they define a deep fake, but the general conclusion is "yes".


2. "Can you handle audio and video?" Yes, but the evaluation process is a lot harder than pictures. Quality means everything, and a high quality video is usually lower quality than a low quality picture. So while the answer is technically "yes", the caveats multiply quickly.


3. "Can you handle documents?" This is where I begin to cringe. By "documents", they almost always mean PDF. Yes, my code can evaluate PDF files and it can often identify indications of edits. However, distinguishing real from AI or identifying specific edits is significantly harder to do. (If you thought evaluating pictures and videos was hard, wait until you try PDF files.)

Over the last 30 years, I have been developing a suite of tools for evaluating PDF documents. Last year I focused on reducing some of the complexity so they would be easier for non-programmers to use. Right now, I have a couple of commercial clients beta-testing the new PDF analyzers -- and they are thrilled! (As one of them remarked, "Christmas came early! Thank you.")

Anything But Typical

When evaluating any kind of media, a quick way to determine if something could be potentially altered or intentionally edited is to look for deviations from the typical case. If an atypical attribute only comes from an edit, then you can detect an edit. Often, you can detect that "something was edited" without viewing the visual content, like by evaluating the file structure.

For example, consider JPEG files. JPEGs have lots of encoding options, including the selection of quantization tables, Huffman encoding tables, and data storage (modes of operation).

The most common JPEG encoding mode is baseline. Baseline includes one set of quantization and Huffman tables, converts colors from RGB to YUV, and groups them in grids. Each grid (MCU, or Minimum Coding Unit) contains one set of YUV components. The baseline method stores MCUs in a raster (left to right, top to bottom), like YUVYUVYUV (or YYYYUVYYYYUV). But these are all within the range of "normal". At FotoForensics, over 85% of JPEGs use baseline encoding.

The next most common encoding is called progressive. It stores the image in layers, from low quality to high quality. (If you've ever seen a web page where an image first appears blurry and then gets sharper and sharper over a few seconds, that's almost certainly progressive encoding.) These account for nearly 15% of the JPEGs that FotoForensics receives; the vast majority of cameras do not use progressive mode. (If you see progressive mode, then it's not direct from a camera.)

If you read the JPEG specifications, they define 12 different encoding modes. I think I have less than a dozen examples of "Lossless JPEG" and "Non-interlaced sequential". The other eight modes appear to be completely theoretical and not used in practice.



When someone talks about a "typical JPEG", they usually mean baseline, but they could mean progressive. However, if you encounter any other type of JPEG, then it's atypical and should immediately trigger a deeper investigation.

The PDF Structure

Detecting PDF edits and expectations requires understanding the required file structure. Unfortunately with PDF, almost nothing is typical, and that's the core forensic problem.

At a very high level, every PDF has some required components:

• Version. The first line is a comment with the minimum version number.


• Objects. A set of objects. Each object has a unique identifier. Objects may define content, pages, metadata, or even other objects. These are the essential building blocks for the PDF document.


• Xref. One or more cross-reference tables (xref or XRef) that identify the location of every object in the file. (These may also have references, creating a chain of cross-reference tables.)


• EOF. The last line needs to be a special "%%EOF" comment, denoting the end of the file.


• Starting Xref. Immediately before the "%%EOF" there must be a "startxref" field with an absolute offset to the top-level cross-reference table. Depending on the PDF version, this could be an older/legacy (PDF 1.0 - 1.4) "xref" object, or a newer/modern (PDF 1.5 or later) XRef object. (Even though xref tables are considered legacy structures, they are more common then the newer XRef objects, even when using newer PDF versions.)


• Starting Object. With the older xref object, there's also a "trailer" object that identifies the top-level object (root, catalog) and where it is located. With XRef, the object's dictionary contains the same type of information.

To render the PDF, the viewer checks the version, finds the "%%EOF", searches backwards for the startxref, then jumps to the cross-reference table. The initial table (xref table with trailer or XRef object) identifies where the starting object is located. (All of the random file access is very much like a Choose Your Own Adventure book. There are even conditionals that may activate different objects.)

I have a collection of over 10,000 PDF files and nearly all have odd corner cases. You might think that being an ISO standard (ISO 32000) would make PDF files more consistent. But in practice, very few applications adhere strictly to the standard.

As an example of PDF's many problems that impact forensic analysis, just consider the issues around versioning, edits, revised object handling, non-standard implementations, and bad pointers.

PDF Versions

With PDF files, lines beginning with a "%" are comments. (This is in general, but comments cannot appear between dictionary fields and values, or in some other parts of the PDF file.) Typically, comments can be ignored, except when they can't.



For example, the very first line of the PDF is a special comment that isn't treated like a comment: it lists the PDF version, like "%PDF-1.5". (I'm going to pick on PDF 1.5, but these problems exist with every PDF version.)

The PDF version does not mean that the PDF complies with the PDF 1.5 version of the specification. Rather, it means that you need a PDF viewer that supports 1.5 or later. Unfortunately, many PDF generators get it wrong:

• Some files may specify a PDF version that is beyond what is really required. For example, it may say "%PDF-1.5" but contains no features found in 1.5 or later versions of the PDF spec. I have many PDF files that specify 1.5 or later, but that could easily be rendered by older viewers.


• Some files may specify the wrong minimal requirements. For example, I have some PDF files that say "%PDF-1.5" but that contain features only found in PDF 1.6 or 1.7.

If a PDF's header claims a version newer than what the viewer supports, the software may refuse to open it -- even if the contents are all completely supported by the viewer. Conversely, if the header version is too old, the viewer might attempt to render it and fail, or worse, render the contents incorrectly.

Does having a particular PDF version or mismatched version/feature indicate an edit? No. It just means the encoder didn't consider version numbers, which is typical for PDF files. This isn't suspicious or unusual.

Seeing Edits

Unfortunately, there is no "typical" or common way to edit a PDF file. When someone alters a PDF, their encoding system may:

• Append after EOF. After the "%%EOF", they may add more objects, cross-reference tables (xref or XRef), another startxref, and a final %%EOF. This results in multiple %%EOF lines: a clear indication of an edit.


• Append after startxref. Some editors remove the last %%EOF, then add in the objects, xref/XRef, startxref, and final %%EOF. This results in only one %%EOF, but multiple startxref entries.


• Replace ending. Some editors remove the final startxref and %%EOF before inserting the new objects and new ending. Seeing multiple xref/XRef tables either indicates an edit or a pipelined output system that stores objects in clusters.


• Revise objects. Revised or obsolete objects may be removed, while new objects may be inserted before rewriting the ending. Seeing unused objects or duplicate objects is an indication of an edit.


• Rewrite everything. Some PDF encoders don't keep anything from the original. They just create a new PDF file. Even with a clean rewrite, this often has artifacts that carry over, indicating that an edit took place.

If you see an edited PDF document, then it means it was clearly and intentionally altered. Right? Well, wrong. As it turns out, most PDF files contain edit residues. Just consider something like a utility bill that is saved as a PDF:

1. Someone created the initial template. This could be a Doc, TeX, PDF, or other file format.


2. Some process filled in the template with customer information (e.g., utility fees and previous payments).


3. Maybe another process also fills in the template with more customer information (e.g., mailing address).


4. Eventually a PDF is generated and sent to the end user.


5. Some enterprise security gateway or "safe attachment" systems (like MS Defender for Office 365, or Proofpoint) may re-encode PDF attachments before delivering it to the user. (The re-encoding usually strips out macros and disables hyperlinks.) Older mobile devices may not download the full PDF immediately; instead, they might request a transcoded or optimized version for preview on the device. In these cases, the user may not receive the byte-per-byte identical copy of what was sent.


6. Eventually the user does receive the PDF. Some users know how to save the PDF directly without any changes, but other users may use "print to PDF" or some kind of "Save" feature that re-encodes the PDF. (Don't blame the user, blame the app's usability. As an aside, I also see lots of people open the camera app and take a screenshot of the camera app's preview screen rather than pressing the big "take photo" button.)

Every step in this pipeline can leave residues that appear in the final PDF. This type of PDF step-wise generation is the norm, not the exception, and every company does it differently. There's nothing typical other than everything being atypical.

But maybe you can compare your suspect utility bill with a known-real utility bill from the same company? Nope, I've seen the same company generate different PDF formats for the exact same utility bill. This can happen if the company has multiple ways to generate a PDF file for the end customer.

Object Identifiers

The PDF file is built using nested objects. The "catalog" object references a "pages" object. The "pages" object references one or more "page" objects. Each "page" object references the contents for the page (fonts, text, images, line drawings, etc.).


(This image shows a sample of typical PDF object nesting.)

Every object has a unique identifier and a revision (generation) number. If you view the binary PDF file, you'll see text like "123 0 obj ... endobj", which identifies object #123 generation 0. If you edit a PDF and replace an object, the spec says that you should update the generation number. In practice, nearly all PDFs use generation number 0 for every object.

Seriously: of the over 10,000 sample PDF documents in my collection, 12 contain objects with non-zero generations. In contrast, it's very common to see the same object number redefined multiple times with the same generation number (123 0 ... 123 0). It may not comply with the specs, but it's typical for PDF generators to always use generation zero. (If you ever see a non-zero generation number, it's technically compliant but definitely atypical; it should set off alarms that the file was edited.)

Bad References

Adobe Acrobat is cited as the authoritative source for following the PDF standard. And that's true... until it isn't.

I have a handful of PDF files that have invalid startxref pointers, but that render fine under Acrobat. For example, there's a popular PDF generator called iText. (As iText describes it, they are the "world's 4th most widely used PDF generator" and "the source of 8% of the world’s PDF documents.") They had a bug where they could generate a startxref file with a bad pointer. (I say "had" because it was fixed in 2011 with version 5.1.2, but lots of people still use older buggy versions.)

Adobe has had 30 years of dealing with bad PDFs. Empirically, Acrobat appears to have special rules such as: "If the offsets are invalid and the file identifies a known generator, then search for the correct offset." As a result, Acrobat will properly render some PDF files that have bad references, but not others.

PDF viewers like Evince (Linux), Firefox, and Chrome are more forgiving. Regardless of the generator: they see the bad references and search for the correct objects. They can often display PDF files that Acrobat cannot. In general, different PDF viewers may handle the same PDF documents differently -- it all depends on the type of offset corruption and how it was auto-fixed.

EOF EOF

The problem with edit detection based on "%%EOF" markers is even more complicated. I have many PDF examples where the encoder wrote a chunk of non-renderable PDF content, dropped in a "%%EOF", then appended the rest of the document with a new "%%EOF".

To the casual eye, this might look like an edit. But in reality, it's a surprisingly common two-stage writing system. The first "%%EOF" is a red herring.

Typical Problems

All of this goes back to the problem of trying to define a typical PDF document. Between different PDF generators and different PDF creation pipelines, there's virtually no consistency. You can't say that a PDF looks suspicious because it has multiple EOF lines, multiple startxref, inconsistent object enumerations, etc.

There are a few ways to detect intentional PDF edits, like seeing reused object IDs, metadata indicating different edits, or changes more than a few seconds apart. But even then, the question becomes whether those edits are expected. For example, if you're filling out a PDF form, then we'd expect the edits (filled-out form) to happen long after the initial document was created. Seeing a clear indication of an edit may not be suspicious; you must take the context into consideration.

Over the last year, I have been hyperfocused on PDF documents in order to detect intentional edits and AI generation. I can finally detect some forms of AI-generated PDF documents. These are often used for fraud, as in, "ChatGPT, can you generate a sample PDF containing a Kansas electrical utility bill for me, for the amount of $1024.45? List the addressee as ..." or "I need a PDF dinner receipt at Martin's Steakhouse for three people last Wednesday." One crucial tell? Look for the dog that didn't bark, such as a filled-in template without an edit between the template and the personalized information. (AI systems generate the template and populate it at the same time, so there is no intermediate edit.)



(It's relatively easy to evaluate this image and determine that it is AI generated. But evaluating the associated PDF is significantly harder.)

The more complicated file formats are, the more difficult they become to evaluate. Among images, videos, audio files, and other types of media, PDF documents are certainly near the top of the difficulty level.

After three months, I finally completed all of the server upgrades. The last upgrade impacted my biggest services: FotoForensics and this blog. (Yeah, I'm also surprised that this blog is so popular. But my logs report over 100,000 regular readers.)

The First Move

Servers run operating systems, and mine were approaching end-of-life. Since I need updated security patches, that means updating the operating systems. And with updated operating systems comes unexpected code changes.

I have over a dozen web services, and I've been carefully upgrading each one. All of the smaller ones were completed at the beginning of December. That just left the larger services: FotoForensics and this blog.

I used the smaller services to work out the migration process and benchmark the upgrade times. For example, Hintfo doesn't need much more than an operating system and a small cache for temporarily holding files. It can easily run on a 25GB disk allocation, but I over-allocated it as a 100GB drive. Copying the 100GB drive (with compression enabled) from the staging system to the production server took about 9 minutes over a dedicated gigabit connection. It took another few minutes to ensure all of the dependencies were installed. (Usually nobody uses Hintfo late a night, so I knew when it was safe to do the update. The 9 minutes of downtime went unnoticed.)

But FotoForensics? It's a two-terabyte drive and used 24/7 by people around the world. I estimated that it would take about five hours to transfer the data from the production server to the staging system.

I didn't want over five hours of downtime, so I came up with a combination of solutions to reduce the overall outage to about 2 minutes:

1. About a month before the move, I began archiving all of the content (anything not used in the last 90 days) to an external RAID. This reduced the in-use disk space to about 300GB.


2. Rather than copying the old server to the new staging environment, I created a new operating system with a 600GB drive (technically, an LVM partition). I zeroed out the entire drive before installing the operating system. Eventually I would need to copy this entire drive to the new production service. By initializing the drive with zeros, the file transfer would hit long runs of zeros and compress them efficiently for a faster transfer.


3. I configured and hardened the new operating system, then I copied over only the files that I needed from the old computer. The net result was about 300GB on a 600GB drive. (Why the extra space? If there was a sudden burst of activity on FotoForensics during this multi-day move, I'd need extra storage.) As a background copy, no downtime was needed. (The copy took over three hours.)


4. After transferring the data, I ran the regression tests to make sure the system was running and stable. This involved some last minute patches, more regression tests, and then redoing the entire move again to make sure I captured every step in my scripts.


5. With a stable, working server ready to go, I transferred all traffic from the old production system to the new staging system the easy way: I changed the IP address on the new system to match the old, and changed the IP address on the old to something different. This way, there is no network address conflict and users wouldn't even know that they switched servers.


6. Between the time when I transferred all of the files to the new server and when I was ready to swap IP addresses, more files arrived on the old service. I temporarily stopped all uploads to FotoForensics (preventing any significant changes to the file system) and performed a final "update" file transfer. The update only pulled any new files that were added or changed since the last transfer. This update took about a minute to complete. Then I re-enabled uploads at the new FotoForensics server and shut down the old server.

I know, this sounds like a lot of steps. (It's actually much more complicated that this.) But this is why I spent months working on the upgrade path and scripting everything. To external users, FotoForensics was "down" for about 30 seconds (late at night during low traffic, so few people noticed), and uploads were disabled for about 2 minutes. Behind the scenes, there was a massive amount of data being transferred between the old and new servers. With system testing and validation, the move took nearly a week.

Moving Back

This big data transfer marked the half-way point of this upgrade process. It allowed me to move everything off the old server and onto a temporary (staging) server with a new operating system. This also gave me time to make sure all of the software was working correctly. (If there was a problem at any point, I could easily turn on the old server and move everything back in under a minute.) However, I still needed to put this newly upgraded server back into the production environment.

I reinstalled the old production server. This included a BIOS upgrade, new operating system, and thorough dusting of the hardware. (Even in the clean room, there's dust.) Then I had to move the virtual machine (which hosts FotoForensics and the Hacker Factor blog) back to the production environment. This is the step that I was dreading, because it really did mean a long period of downtime.

Keep in mind: I could have performed the same file transfer process in reverse. However, it took time to make sure I had the entire server installed and configured before the file transfer, and I didn't want to miss a step. The safer solution was to shutdown the virtual system and transfer the entire (new) 600GB virtual drive back to the production environment.

Even with the reduced size of 600GB, the transfer could still take hours. I had considered other options, but each had undesirable complexities. For example, I could take down the server, perform an LVM snapshot, and bring the server back up. The total downtime would be about 30 seconds. Then I could transfer the main drive at my convenience and transfer any changes after the snapshot separately. However, this has a lot of complexities since FotoForensics is constantly writing cache files. It's also a fragile solution since a reboot could generate a few hundred gigs of post-snapshot changes.

I ended up taking a much simpler approach, even through it meant a longer downtime:

1. I created a tiny virtual machine that just hosts a static web page saying that the server was down for scheduled maintenance. I put it on the same IP address as the production server and moved the production server off that external IP address. The result was that users suddenly saw a "Down for scheduled maintenance, be back soon" message rather than a server timeout error.
   
   


2. I decided that the simplest and safest option was to transfer the entire 600GB drive over the dedicated backbone. I estimated that it would take 90 to 120 minutes. It ended up taking 82 minutes (1 hour 22 minutes). That's 2 minutes of manual labor and 1 hour 20 minutes of completely automated scripts. (During that time, I was panic-waiting like an expectant father, just glued to the logs and watching the progress status.)


3. To reduce the impact on users, I decided to do it late at night on Christmas (Dec 25). FotoForensics has been around since 2012, so I have lots of historical data telling me that it should be a low-volume period. During the 82-minute downtime, about 30 users saw the message -- excluding me as I repeatedly tested to see if the message was working.


4. The transferred drive was 600GB, but I need 2 terabytes on the production system. (Really, 1TB should be fine, but I believe in overallocating resources.) With Linux, you can resize a hard drive, but it's a multi-command process. Fortunately, I had already turned it into a script. I simply run the command, enter the VM's name, and specify the desired disk size. The script performed all of the complicated mount, unmount, resize, and file system checks. After the data transfer, increasing the drive from 600GB to 2TB took less than a minute.

With the transfer completed and drive resized, I started the new production system. After a brief set of tests, I finally moved over the IP address from the staging system to the production server. Amazingly, everything just started working. As much as these server transfer scripts were hard to create, they really made the final process painless.

Going Sideways

I expected to have problems with the scripts. This is why I spent months creating and testing them. What I didn't anticipate was a distributed denial-of-service (DDoS) attack while I was part-way through a big update. (Fortunately, the data transfer was over the internal network, so the attack had no impact.)

This attack, on Dec 19, was another UDP flood, but it wasn't as large as the previous North Korea attacks; this DDoS only consumed about 95% of my bandwidth. (The North Korea attacks from 2024 consumed all of my bandwidth and took down a large portion of Northern Colorado, Wyoming, and a few other states.)

Previous attacks went after my web sites. This time, the attacker targeted my mail server. Why the mail server? My web services are now hidden behind CloudFlare, so it's hard to find them. (They can attack CloudFlare -- who offer free DDoS protection -- but they won't find my servers.) But a mail server must be listed in the DNS MX records.

As a technical note: CloudFlare can proxy web servers because the browser connects and says "I want to talk to hackerfactor.com" (or whatever server name they are proxying for). But email servers don't require a server name. The client would connect to the mail server at CloudFlare and CloudFlare wouldn't know where to forward the connection.

Because of the previous attacks, I had moved all of my other servers to a different network range. However, I had left my mail server on the old range since it was still exposed. Unfortunately, I still use all of my own routing equipment. The incoming flood against the mail server slowed down access to all of my other services.

My workaround? I decided to bite the bullet and pay Google $7/month for Google Workspace. The Google Workspace service includes email relaying. Google receives the email and forwards it to my mail server. Moreover, all of my outgoing email relays through Google. Now if someone wants to DDoS my mail server, they will need to target Google. (I'm fine with that.)

On top of this, I moved my mail server to a new IP address and restricted who can access it. Since all incoming email should be relayed by Google to me, I have new iptables rules that only permit Google to connect to my mail server. With this change, the attacker cannot even scan for me.

Configuring Google Workspace for email relaying was (mostly) painless. Typically I avoid using AI to help with technical problems. (Vibe programming sucks, and for diagnosing problems, most AI systems end up exploring incredibly complex and fragile solutions.) However, Google Gemini is trained on Google's services. I described my configuration and it provided (correct) step-by-step instructions. The hardest part was identifying any bugs and figuring out how to phrase the questions correctly.

There was one other very unexpected aspect to this change: spam. Running my own mail server, I was receiving at least a dozen spam messages per day. With Google, I'm seeing ZERO spam. It isn't that Google is filtering spam for me. (My spam folder at Google is empty and their logs show 3-4 blocked messages a day.) Rather, it seems that spammers see that Google is the mail server and they don't even try to send.

Always an Upside

My opinion about DDoS attacks hasn't changed: I still think it's a coward's attack. If the goal is to punish me for something I wrote or to make me reconsider my life choices, then they failed. Since I don't know who is attacking me or why, I have no way to know what behavior they want me to change.

If their goal was to make me aware of a server weakness, then they could have easily sent me an email. Trying to knock my server offline isn't as effective.

On the other hand, my servers get hit by a big DDoS attack every 1-2 years. (2019 and 2024 were the most notable ones.) Each time, I've not only survived the attack with nothing more than a minor inconvenience, but I figured out new ways to prevent the attacks. (And I share this information with my customers, making them more secure, too.)

What does a DDoS do? It helps me identify my limitations and permits me to create a much stronger service. (Of course, I'd rather do this on my own schedule instead of in response to a fire drill, but as a system administrator, you have to be flexible.)

Up, Up, and Away!

The new servers with updated operating systems are up and running, while the new mail server appears to be better hidden from attack. Surprisingly, everything is benchmarking much faster!

• On the same hardware, FotoForensics is about 8% faster at processing media. I'm just assuming that the old operating system was less efficient or contained unnecessary bloat that I stripped out this time. (I've also learned how to make it even more efficient next time I have to go through this process.)


• Besides having a better-hidden mail server, I'm seeing less spam and receiving email much faster. My self-tests from an external account to Gmail to me used to take 1-2 minutes. Now it takes seconds. I think this is because Google Workspace removed a lot of the internal relaying, making the path much more direct.


• While updating the server BIOS, I enabled some power efficiency settings in the hardware. The rack is using about 12% less power than before the upgrades.

This year, I've been heavily focused on efficiency and savings. This includes optimizing my solar-powered battery system, heating and cooling cost reductions, and making better use of older equipment (I really do like my new old keyboard). With everything running faster and more efficiently, I'm ready to head into 2026. Happy New Year.

I've been battling with an operating system problem for the last few months. The problem? The operating system on some of my servers is screaming toward "end of life". That means they need to be updated.

Previously, I'd updated each server separately and taken notes as kind of an installation script. Of course, those scripts are great for notes but ended up not working well in practice. But at least I knew what needed to be installed.

This time I had the idea of actually scripting everything. This is particularly important since I'll be updating three servers, each with a handful of virtual machines -- and they all need to be updated. (Well, a few don't need to, but for consistency, I want to make them all the same.) The scripts should allow the migration to be more rapid and consistent, without depending on my memory or lots of manual steps.

There are a lot of steps to this process, but each step is pretty straightforward:

1. Choose a new operating system. (I decided on Ubuntu 24.04 LTS for now.)


2. Install the base operating system as a minimal server. Customize it to my liking. (E.g., I have some shell aliases and scripts that I use often and they need to be on every server. I also need to harden the basic OS and add in my custom server monitoring code.)


3. Install a hypervisor. In virtual machine terminology, the hypervisor is "dom0" or the "host". It runs one or more virtual machines (VMs). Each VM is often called a "guest" or "domu". I have 3 production servers and a 4th "hot backup" in case of hardware failures or for staging migrations, so I'll be installing 4 dom0 systems and a bunch of domu on each dom0.


4. Create a template virtual machine (template domu) and configure it with my defaults.


5. I'll be updating the servers, one at a time. For each virtual machine (VM) on the old server:
   
   1. Copy the template on the staging server to a new VM.
   2. Transfer files from the old VM on the old server to the new VM on the staging server.
   3. Make sure it all works.


6. When the staging server has everything running and the old server is no longer in use:
   
   1. Reinstall the old server using the installation scripts.
   2. Transfer each new VM from the staging server to the production server.
   3. Make sure it all works.


7. When everything has been transferred and is running on the production server, remove it all from the staging server and then start the same process for the next old server.

It's a lot of steps, but it's really straightforward. My installation scripts have names like:

install-step00-base-os.sh
install-step01-user.sh
install-step02-harden.sh
install-step03-network.sh
install-step04-ufw.sh
install-step05-create-dom0.sh
install-step06-system-monitor.sh
install-step10-domu.sh
install-step11-domu-user.sh
install-step12-domu-harden.sh
install-step13-domu-network.sh
install-step14-domu-ufw.sh
install-step20-migration-prep.sh
install-step21-make-clone.sh
install-step22-copy-old2clone.sh
install-step23-validate.sh
install-step24-guest-move.sh
install-step25-guest-cleanup.sh

I expected this entire process to take about a month. In reality, I've been battling with every step of the process for nearly 3 months.

The first problems

I really thought choosing an operating system and a hypervisor was going to be the easiest choice. I had previously been using Xen. Unfortunately, Xen is not well-supported under Ubuntu 24.04. (Ubuntu with Xen refused to boot. I'm not the only person with this problem.)

Since Ubuntu 24.04 has been out for over a year, I'm not going to hold my breath for a quick fix. I decided to switch to KVM -- it's what the Debian and Ubuntu developers use. KVM has a lot of really nice features that Xen is missing, like an easy(?) way to move existing VMs between servers.

However, I absolutely could not get IPv6 working under KVM. My ISP doesn't sell fixed IPv6 ranges. Instead, everyone uses DHCPv6 with "sticky" addresses (you get an address once and then keep it).

I should have known that DHCPv6 would be a problem with Ubuntu 24.04: during the base Ubuntu OS install, it failed to acquire an IPv6 address from the installation screen. IPv4 works fine using DHCP, but IPv6 does not. Part of the problem seems to be with the OS installer.

However, I'm sure part of the problem is also with my ISP. You see, with IPv4, there's one way to get a dynamic address. However, IPv6 never solidified around a single method. For example:

• DHCPv6 vs SLAAC: DHCPv6 provides stateful configurations, while SLAAC is for stateless. The ISP may even use a combination of them. For example, you may use DHCPv6 for the address, but SLAAC for the routes.


• Addressing: There are options for acquiring a temporary address, prefix delegation, and more. (And if you request a prefix delegation but provide the wrong mask size, then it may not work.)


• Routing: Even if you have the address assigned, you may not have a route until the ISP transmits an IPv6 router advertisement (RA). How often those appear depends on the ISP. My ISP transmits one RA every 10-20 minutes. So even if you think everything is working, you might need to wait 10-20 minutes to confirm that it works.

After a week of fighting with the IPv6 configuration, I managed to get DHCPv6 working with my ISP for a /128 on dom0, but I could never get the /56 or virtual servers to work.

While debugging the dom0 issues, I found a problem with KVM's internal bridging. I have a network interface (let's call it "wan"). All of the VMs access it over a bridge (called "br-wan").

• Under Xen, each VM uses a dynamically allocated tap that interfaces with the bridge. The tap relays the VM's MAC address. As a result, the ISP's DHCPv6 server sees a request coming from the virtual system's MAC address and allocates an address associated with the MAC. This allowed IPv6 to work under Xen.


• KVM also has a virtual tap that accesses the bridge, but the tap has a different MAC address than the VM. (This isn't a bug; it's just a different architectural decision that the KVM developers made.) As a result, the DHCPv6 server sees a request for an address coming from the tap's MAC, but the confirmation comes from the VM's MAC address. Since the address changed, the confirmation fails and the machine never gets an IPv6 address. (I could not find a workaround for this.)

I spent over a month battling with the IPv6 configuration. I am finally convinced that none of the KVM developers use IPv6 for their VMs, or they use an ISP with a less hardened DHCPv6 configuration. Since I'm up against a hard deadline, none of my new servers will have IPv6 enabled. (I'm still using CloudFlare for my front-end, and they support IPv6. But the connection from CloudFlare to me will be IPv4-only.)

How far did I get with IPv6?

• dom0 can consistently get a /128 (that's ONE IPv6 address) if I clone the NIC's hardware address to the bridge.


• With a modified configuration, the dhclient process on dom0 can request and receive a /56 from the ISP, but then the ISP refuses to confirm the allocation so dhclient never accepts it (because the MAC changes).


• Switching from the default bridge to a macvtap makes no difference.


• Flushing old leases, changing how the DUID is generated, creating my own post-dhcpv6 script to accept the allocation... these all fail.


• While dom0 could partially work, my VM guest systems never worked. They were able to request and receive an allocation, but the confirmation was never accepted.

The lesson? If it doesn't work straight out of the box, then it doesn't work.

As one of my friends put it: "IPv6 is the way of the future, and it always will be." It's really no wonder that IPv6 hasn't had wider adoption over the last 30 years. It's just too complicated and there are too many incompatible configurations. (Given all of the problems I've encountered with virtual machines, I now understand why many cloud providers do not support IPv6.)

Cloning Templates

Once IPv6 was off the table, I turned my attention toward creating a reproducible VM template. Configuring a KVM domu/guest virtual server was relatively painless.

The idea is that I can clone the template and configure it for any new server. The cloning command seems relatively painless, unless you want to do something a little different.

For me, the template uses a QCOW2 file system. However, the running configured guest servers all use the Logical Volume Management (LVM) system. I allocate the logical volume (lvcreate) and then clone the template into the new LVM disk. (sudo qemu-img convert -p -f qcow2 -O raw "$SOURCE_QCOW2" "$LV_PATH")

The good news is that this works. The bad news is that the template is only a 25G disk, but the logical volume is allocated as 200G -- because the final server needs more disk space. If I boot the cloned system, then it only see a 25G hard drive. Expanding the cloned image to the full disk size is not documented anywhere that I could find, and it is definitely complicated. Here's the steps that I finally found that work:

# Create the new server's disk space

sudo lvcreate -L "$GuestVGsize" -n "$GuestName" "$GuestVG"

LV_PATH="/dev/$GuestVG/$GuestName"



# Find the template's disk. My template domu is called "template".

SOURCE_QCOW2=$(virsh dumpxml template | grep 'source file' | awk -F\' '{print $2}')



# Do the actual cloning (with -p to show progress)

sudo qemu-img convert -p -f qcow2 -O raw "$SOURCE_QCOW2" "$LV_PATH"



# LV_PATH is a single volume that contains multiple partitions.

# Partition 1 is the bootloader.

# Partition 2 is the file system.

# Get the file system's partition path

LV_CONTAINER_PARTITION_NAME=$(sudo kpartx -l "$LV_PATH" | tail -n 1 | awk '{print $1}')

LV_CONTAINER_PARTITION="/dev/mapper/${LV_CONTAINER_PARTITION_NAME}"



# Get the starting sector for resizing the 2nd (data) partition.

START_SECTOR=$(sudo gdisk -l "$LV_PATH" | grep '^ *2' | awk '{print $2}')

sudo kpartx -d "$LV_PATH"

sleep 2 # wait for it to finish



# Edit the partition table to expand the disk

sudo sgdisk "$LV_PATH" -d 2

sudo sgdisk "$LV_PATH" -n 2:"$START_SECTOR":0 -c 2:"Linux Filesystem" -t 2:8300

sudo sgdisk "$LV_PATH" -w

# Inform the operating system of this change

sudo partprobe "$LV_PATH"



# Extract links to the partitions for cleanup

sudo kpartx -a "$LV_PATH"

# Check the file system

sudo e2fsck -f "$LV_CONTAINER_PARTITION"

# Resize it

sudo resize2fs "$LV_CONTAINER_PARTITION"



# If you want: mount $LV_CONTAINER_PARTITION and edit it before the first boot.

# Be sure to umount it when you are done.



# Done! Remove the partition links

sudo kpartx -d "$LV_PATH"

This took me a few days to figure out, but now the cloned guest has the correct disk size. I can now easily clone the template and customize it for specific server configurations. (I skipped over the steps related to editing the KVM's xml for the new server or using virsh to activate the new cloned image -- because that would be an entire blog post all by itself.)

Copying Files

Okay, assume you have a working KVM server (dom0) and an allocated new server with the right disk size. Now I want to copy the files from the old server to the new server. This is mainly copying /home, /etc/postfix, /etc/nginx, and a few other directories. Copying the contents should be easy, right?

'rsync' would be a great option. However, I'm copying from a production server to the pre-deployment environment. Some of the files that need to be transferred are owned by other users, so the rsync would need to run as root on both the sender and recipient systems. However, my servers do not permit logins as root. This means that I can't rsync from one server to another.

'tar' is another great option. In theory, I could ssh into the remote system, tar up the files and transfer them to the new guest server. However, to get the files, tar needs to run as root on the production server. (We permit 'sudo', but not direct root logins.) An ideal solution would be like:

ssh prodserver "cd / ; sudo tar -cf - home" | (cd / ; sudo tar -xvf - )

Unfortunately, this approach has a few problems:

• sudo requires a terminal to get the password. That means using "ssh -t" and not just "ssh".


• The terminal receives a text prompt. That gets fed into the decoder's tar command. The decoder says "That's not a tar stream!" and aborts.

I finally worked out a solution using netcat:

On the receiving server:
cd / ; nc -l 12345 | tar -xvf - This waits for a connection on port 12345 and sends the data to tar to extract. netcat will terminate when the stream ends.

To get to the production server:
ssh -o LogLevel=error -t "$OldServer" "cd / ; sudo bash -c 'echo Running as sudo' ; sudo tar -cf - $GetPathsReal | nc newserver 12345

This is a little more complicated:

• "-o LogLevel=error" My production ssh server displays a banner upon connection. I need to hide that banner so it doesn't confuse tar.


• "-t" opens a terminal, so sudo will prompt for a password.


• "sudo bash -c 'echo Running as sudo'" Get the sudo prompt out of the way. It must be done before the tar command. This way, the next sudo call won't prompt for a password.


• "sudo tar -cf - $GetPathsReal | nc newserver 12345" This tars up the files that need to be transferred and sends them through a netcat tunnel.

The rsync solution would be simple and elegant. In contrast, using tar and netcat is a really ugly workaround -- but it works. Keep in mind, the netcat tunnel is not encrypted. However, I'm not worried about someone in my internal network sniffing the traffic. If you have that concern, then you need to establish an encrypted tunnel. The catch here is that ssh does not transfer the tar stream -- the tar stream comes over a parallel connection.

Current Status

These are far from all of the problems that I had to resolve. After nearly 3 months, I finally have the first three VMs from one dom0 migrated to the new OS. Moreover, my solution is 95% scripted. (The remaining 5% is a human entering prompts and validating the updated server.) Assuming no more big problems (HA!), it will probably take me one day per VM and a half-day per server.

The big lessons here?

• With major OS migrations, expect things to break. Even common tasks or well-known processes are likely to change just enough to cause failures.


• Automating this process is definitely worth it. By scripting every step, I ensure consistency and an ability create new services as needed. Moreover, some of the lessons (like battling with IPv6, fighting with file transfers, and working out how to deal with logical volumes) were needed anyway; those took months to figure out and the automated scripts document the process. Now I don't have to work it out each time as a special case.


• After nearly 30 years, IPv6 is still much less standardized across real-world environments than people assume.


• KVM, templates, and logical volumes require more knowledge than typical cloud workflows lead you to expect.

This process took far longer than I anticipated, but the scripting investment was worth it. I now have a reproducible, reliable, and mostly automated path for upgrading old systems and creating new ones.

Most of my running services will be moved over and nobody will notice. Downtimes will be measured in minutes. (Who cares if my mail server is offline for an hour? Mail will just queue up and be delivered when it comes back up.) Hintfo and RootAbout will probably be offline for about 5 minutes some evening. The big issue will be FotoForensics and Hacker Factor (this blog). Just the file transfers will probably take over an hour. I'm going to try to do this during the Christmas break (Dec 24-26) -- that's when there is historically very little traffic on these sites. Wish me luck!

Over the summer and fall, SEAL saw a lot of development. All of the core SEAL requirements are now implemented, and the promised functionality is finally available!

SEAL? What's SEAL?

I've written a lot of blog entries criticizing different aspects of C2PA. In December 2023, I was in a call with representatives from C2PA and CAI (all from Adobe) about the problems I was seeing. That's when their leadership repeatedly asked questions like, "What is the alternative?" and "Do you have a better solution?"

It took me about two weeks to decide on the initial requirements and architect the framework. Then I started writing the specs and building the implementation. The result was initially announced as "VIDA: Verifiable Identity using Distributed Authentication". But due to a naming conflict with a similar project, we renamed it to "SEAL: Secure Evidence Attribution Label".

C2PA tries to do a lot of things, but ends up doing none of it really well. In contrast, SEAL focuses on just one facet, and it does it incredibly well.

Think of SEAL like a digital notary. It verifies that a file hasn't changed since it was signed, and that the signer is who they say they are. Here's what that means in practice:

• Authentication: You know who signed it. The signer can be found by name or using an anonymized identifier. In either case, the signature is tied to a domain name. Just as your email address is a unique name at a domain, the SEAL signer is unique to a domain.


• No impersonations: Nobody else can sign as you. You can only sign as yourself. Of course, there are a few caveats here. For example, if someone compromised your computer and steals your signing key, then they are you. (SEAL includes revocation options, so this potential impact can be readily mitigated.) And nothing stops a visually similar name (e.g., "Neal" vs "Nea1" -- spelled with the number "1"), but "similar" is not the same.


• Tamper proof: After signing, any change to the file or signature will invalidate the signature. (This is a signficantly stronger claim than C2PA's weaker "tamper evident" assertion, which doesn't detect all forms of tampering.)


• No central authority: Everything about SEAL is distributed. You authenticate your signature, and it's easy for a validator to find you.


• Privacy: Because SEAL's authentication information is stored in DNS, the signer doesn't know who is trying to validate any signature. DNS uses a store-and-forward request approach with caching. Even if I had the ability to watch my own authoritative DNS server, I wouldn't know who requested the authentication, why they were contacting my server (DNS is used for more things than validation), or how many files they were validating. (This is different from C2PA's X.509 and OCSP system, where the certificate owner definitely knows your IP address and when you tried to authenticate the certificate.)


• Free: Having a domain name is part of doing business on the internet. With SEAL, there is no added cost beyond having a domain name. Moreover, if you don't have a domain name, then you can use a third-party signing service. I currently provide signmydata.com as a free third-party signer. However, anyone can create their own third-party signer. (This is different from C2PA, where acquiring an X.509 signing certificate can cost hundreds of dollars per year.)

One Label, Every Format

SEAL is based on a proven and battle-tested concept: DKIM. Virtually every email sent today uses DKIM to ensure that the subject, date, sender, recipients, and contents are not altered after pressing "send". (The only emails I see without DKIM are from spammers, and spam filters rapidly reject emails without DKIM.)

Since DKIM is good enough for protecting email, why not extend it to any file format? Today, SEAL supports:

• Images: JPEG, PNG, WebP, HEIC, AVIF, GIF, TIFF, SVG, DICOM (for medical imaging files), and even portable pixel maps like PPM, PGM, and PNM.


• Audio: AAC, AVIF, M4A, MKA, MP3, MPEG, and WAV. (Other than 'raw', this covers practically every audio format you will encounter.)


• Videos: MP4, 3GP, AVI, AVIF, HEIF, HEVC, DIVX, MKV, MOV (Quicktime), MPEG, and WebM. (Again, this covers almost every video format you will encounter.)


• Documents: PDF, XML, HTML, plain text, OpenDocument (docx, odt, pptx, etc.), and epub.


• Package Formats: Java Archive (JAR), Android Application Package (APK), iOS Application Archive (iPA), Mozilla Extension (XPI), Zip, Zip64, and others.


• Metadata Formats: EXIF, XMP, RIFF, ISO-BMFF, and Matroska.

If you're keeping count, then this is way more formats than what C2PA supports. Moreover, it includes some formats that the C2PA and CAI developers have said that they will not support.

What's New?

The newest SEAL release brings major functional improvements. These updates expand how SEAL can sign, reference, and verify media, making it more flexible for real-world workflows. The big changes to SEAL? Sidecars, Zip support, source referencing, and inline public keys.

New: Sidecars!

Typically, the SEAL signature is embedded into the file that is being signed. However, sometimes you cannot (or must not) alter the file. A sidecar stores the signature into a separate file. For verifying the media, you need to have the read-only file that is being checked and the sidecar file.

When are sidecars useful?

• Read-only media: Whether it's a CD-ROM, DVD, or a write-blocker, sometimes the media cannot be altered. A sidecar can be used to sign the read-only media by storing the signature in a separate file.


• Unsupported formats: SEAL supports a huge number of file formats, but we don't support everything. You can always use a sidecar to sign a file, even if it's an otherwise unsupported file format. (To the SEAL sidecar, what you are signing is just "data".)


• Legal evidence: Legal evidence is often tracked with a cryptographic checksum, like SHA256, SHA1, or MD5. (Yes, legal often uses MD5. Ugh. Then again, they still think FAX is a secure transmission method.) If you change the file, then the checksum should fail to match. (I say "should" because of MD5. Without MD5, it becomes "will fail to match".) If it fails to match, then you have a broken chain of custody. A sidecar permits signing evidence without altering the digital media.

New: Zip!

The most recent addition to SEAL is support for Zip and Zip64. This makes SEAL compatible with the myriad of zip-based file types without introducing weird side effects. (OpenDocuments and all of the package formats are really just zip files containing a bunch of internal files.)

Deciding where to add the signature to Zip was the hardest part. I checked with the developers at libzip for the best options. Here's the choices we had and why we went with the approach we use:

• Option 1: Sidecar. Include a "seal.sig" file (like a sidecar) in the zip archive.
  
  • Pro: Easy to implement.
  • Con: Users will see an unexpected "seal.sig" file when they open the archive.
  Since we don't want to surprise anyone with an unexpected file, we ruled out this option.


• Option 2: Archive comment. Stuff the SEAL record in the zip archive's comment field.
  
  • Pro: Easy to implement.
  • Meh: Limited to 65K. (Unlikely to be a problem.)
  • Con: Repurposes the comment for something other than a comment.
  • Con: Someone using zipinfo or other tools to read the comment will see the SEAL record as a random text string.
  (Although there are more 'cons', none are really that bad.)


• Option 3: Per-file attribute. Zip permits per-file extra attributes. We can stuff the SEAL in any of these and have it cover the entire archive.
  
  • Pro: Easy to implement.
  • Con: Repurposes the per-file attribute to span the entire archive. This conflicts with the basic concept of Zip, where each file is stored independently.


• Option 4: Custom tag. Zip uses a bunch of 4-byte tags to denote different segments. SEAL could define its own unique 4-byte tag.
  
  • Pro: Flexible.
  • Con: Non-standard. It won't cause problems, but it also won't be retained.
  If this could be standardized, then this would be an ideal solution.


• Option 5: Custom encryption field. Have the Zip folks add in a place for storing this. For example, they already have a place for storing X.509 certs, but that is very specific to Zip-based encryption.
  
  • Pro: Could be used by a wide range of Zip-signing technologies.
  • Con: We don't want to repurpose the specific X.509 area because that could cause compatibility problems.
  • Con: There are some numeric codes where you can store data. However, they are not standardized.
  The folks at libzip discouraged this approach.

After chatting with the libzip developers, we agreed that options 1 and 3 are not great, and options 4 and 5 would take years to become standardized. They recommended Option 2, noting that today, almost nobody uses zip archive comments.

For signing a zip file, we just stuff the text-based SEAL signature in the zip archive's comment field. *grin* The signature signs the zip file and all of its contents.

The funny thing about zip files is that they can be embedded into other file formats. (For those computer security "capture the flag" contests, the game makers often stuff zip files in JPEG, MP3, and other files formats.) The sealtool decoder scans the file for any embedded zip files and checks them for SEAL signatures.

New: Source Referencing!

This feature was requested by some CDN providers. Here's the problem: most content delivery networks resize, scale, and re-encode media in order to optimize the last-mile delivery. Any of these changes would invalidate the signer's signature.

With SEAL, you can now specify a source URL (src) for the validator to follow. It basically says "I got this content from here." The signer attests to the accuracy of the remote resource. (And they can typically do this by adding less than 200 bytes to the optimized file.)

Along with the source URL, there can also be a cryptographic checksum. This way, if the URL's contents change at a later date (which happens with web content), then you can determine if the URL still contains the source information. In effect, SEAL would tell you "it came from there, but it's not there anymore." This is similar to how bibliography formats, like APA, MLA, or Chicago, require "accessed on" dates for online citations. But SEAL can include a cryptographic checksum that ensures any content at the location matches the cited reference. (As an example, see the Harvard Referencing Guide. Page 42 shows how to cite social media sources, like this blog, when used as a source.)

As an example, your favorite news site may show a picture along with an article. The picture can be SEAL-signed by the news outlet and contain a link to the uncropped, full-size picture -- in case someone wants to fact-check them.

Source referencing provides a very rudimentary type of provenance. It says "The signer attests that this file came from here." It may not be there at a later date, but it was there at one time.

New: Inline Public Keys!

While Zip impacts the most file formats, inline public keys make the cryptography more flexible and future-proof.

With a typical SEAL signature, the public key is located in DNS. The association with the DNS record authenticates the signer, while the public key validates the cryptography. If the cryptography is invalid, then you cannot authenticate the signer.

With inline public keys, we split the functionality. The public key is stored inside the SEAL signature. This permits validating the cryptography at any time and without network access. You can readily detect post-signing tampering.

To authenticate the signer, we refer to DNS. The DNS record can either store the same public key, or it can store a smaller digest of the public key. If the cryptography is valid and the public key (either the whole key or the digest) exists in the DNS record, then SEAL authenticates the signer.

When should inline public keys be used?

• Offline validation. Whether you're in airplane mode or sitting in a high security ("air gap") environment, you can still sign and validate media. However, you cannot authenticate the signature until you confirm the public key with the DNS record.


• Future cryptography. Current cryptographic approaches (e.g., RSA and EC) use public keys that are small enough to fit in a DNS TXT field. However, post-quantum cryptography can have extremely long keys -- too long for DNS. In that case, you can store the public key in the SEAL field and the shorter public key digest in the DNS record.


• Archivists. Let's face it, companies come and go and domain names may expire or change owners. Data that is verifiable today may not be verifiable if the DNS changes hands. With inline public keys, you can always validate the cryptography, even when the DNS changed and you can no longer authenticate the signer. For archiving, you can combine the archive with a sidecar that uses an inline public key. This way, you can say that this web archive file (WARC) was accurate at the time it was created, even if the source is no longer online.

Basically, inline public keys introduces a flexibility that the original SEAL solution was lacking.

Next Up

All of these new additions are fully backwards-compatible with the initial SEAL release. Things that were signed last year can still be validated with this newer code.

While the command-line signer and validator are complete, SEAL still needs more usability -- like an easy-access web front-end. Not for signing, but for validating. A place where you can load a web page and select the file for validating -- entirely in your web browser and without uploading content to a server. For example, another SEAL developer had created a proof-of-concept SEAL validator using TypeScript/JavaScript. I think the next step is to put more effort in this direction.

I'm also going to start incorporating SEAL into FotoForensics. Right now, every analysis image from FotoForensics is tagged with a source media reference. I think it would be great to replace that with a SEAL signature that includes a source reference. Over the years, I've seen a few people present fake FotoForensics analysis images as part of disinformation campaigns. (It's bound to become a bigger problem in the future.) Using SEAL will make that practice detectable.

While I started this effort, SEAL has definitely been a group project. I especially want to thank Shawn, The Boss, Bill not Bob, Bob not Bill, Dave (master of the dead piano), Dodo, BeamMeUp8, bgon, the folks at PASAWG for their initial feedback, and everyone else who has provided assistance, reviews, and criticisms. It's one thing to have a system that claims to provide authentication, provenance, and tamper detection, but it's another to have one that actually works -- reliably, transparently, at scale, and for free.

Scary stories come in all forms. For system administrators, late night outages and network attacks can cause nightmares.

It's been a year since my last big distributed denial-of-service (DDoS) attack. I had been holding off about blogging about this for a few reasons. First, I wanted to make sure it was over. Second, I didn't want to tip off the attackers about what I learned about them. (I had been quietly telling other people about the attack details. It's even helped some other victims of the same attack.) And finally, I wanted to see if they would come back and trigger any of my traps. (Nope!)

The First Wave

On Wednesday, 23-Oct-2024 at 3pm local time (21:00 GMT), my servers came under a massive distributed denial of service attack.

My servers are physically located in a machine room, about 20 feet away from my office. When my servers come under any kind of load, their fans rev up. Even though they are a room away, I can hear the fans pick up. (It sounds like a jet engine.) When the attack started, I heard two servers ramp up.

My first thought was that one of my customers was probably analyzing videos. That always causes a higher load, but it usually lasts a minute. When the sound continued, I checked the servers themselves. None of the virtual machines had any high-load processes running. In fact, the loads were all hovering around 0.1 (virtually no use). It took me a few moments to find the cause: my server was rejecting a huge number of packets. It was definitely a DDoS attack.

I don't know the exact volume of the attack. My servers were logging a sustained 300Mbps and over 150,000 packets per second. (The logging and packet rejections were enough to cause the fans to ramp up.) However, I'm sure the volume was more than that -- because the upstream router was failing. I later learned that it was even larger: the upstream router to the upstream router was failing. The 300Mbps was just the fraction that was getting through to me. The attacker wasn't just knocking my service offline; they took down a good portion of Northern Colorado. (I grabbed some sample packet captures for later analysis.)

I first confirmed that my servers had not been compromised. (Whew!) Then I called my ISP. They had already noticed since the attack was taking down a few hundred businesses that used the same router.

My ISP did the right thing: they issued a black-hole for my impacted IP address. This droped the traffic long before it reached my server or even the impacted routers.

The First Mitigation

Since the attackers were only going after one IP address, my ISP and I thought I could get away with changing my DNS and moving my impacted services to a different address. On that single IP address, I had a handful of services.

• I first moved FotoForensics. No problem. Usually DNS records are cached for a few hours. Long before any of this happened, I had configured my DNS to only cache for 5 minutes (the minimum time). Five minutes after changing my DNS record, the service came back up and users were able to access it.


• I then moved some of my minor services. Again, after 5 minutes, they were back up and running.


• I could have moved all of my services at once, but I wanted to know which one was being attacked. The last service I moved was this blog. After 5 minutes, the DDoS returned, hitting the new address.

This told me a couple of things:

1. The attack started at precisely 3:00pm and it lasted exactly 12 hours. This appeared to be a schedule attack.


2. They were explicitly targeting my blog and Hacker Factor web service. (Why? What did I do this time? Or maybe, what did I write recently?)


3. They were repeatedly checking DNS to see if I moved. They knew this was a logical step and they were watching for it. That's a level of sophistication that your typical script kiddie doesn't think about. Moreover, it appeared to be an automated check. (Automated? I might be able to use that for a counter attack.)

Looking over the network logs, I saw the packets that were doing the attack:

• It was a flood over UDP. With UDP, you can just shoot out packets (including with fake sender IP addresses) and overwhelm the recipient. This attack varied from targeting port 123/udp (network time protocol) and 699/udp (an unknown port). Neither of these existed on my server. It wasn't about taking down my servers; it was about taking down the routers that lead to my servers.


• Every UDP packet has a time-to-live (TTL) value that gets decremented with each router hop. The TTL values from the attack packets didn't match the sender's address in the UDP packets. That tells me that the sender IP addresses were forged. I run a bunch of honeypots that benchmark attacks year round. The packet TTLs and timings were consistent with traffic coming from Europe and Asia. I then tracked the attack to AS4134 (China).


• They were only attacking over IPv4, not IPv6. That's typical for most bulletproof hosting providers. (These are the types of companies with high bandwidth and no concerns about their customers causing massive network attacks.)


• When the network address was blocked (black hole), the DDoS stopped shortly afterwards. When my DNS changed, the attack restarted. This tells me that they were monitoring my address in order to see when it went down.


• After I changed IP address, I noticed something. Buried in the logs was a single IP address at a university (not in China). It was continually polling to see if my server was up. Blocking that one IP address caused the DDoS against the new IP address to turn off. The attackers appeared to be using this as a way to decide when to disable the attack. (Finding this needle in the 150,000 packets-per-second haystack was the hard part.)

All of this tells me the how, but not the who or why.

Who and Why?

I turned the IP addresses, packet captures, and logs over to some of my, uh, friends. I do not know the details of their methods, but they are very effective.

• They tracked the bulk of the DDoS attack to servers often associated with attacks from North Korea.


• They found that the university system was in a specific university lab. The lab members mostly had Korean names. We suspect that either (A) at least one of the students was North Korean posing a South Korean, or (B) one of the students had downloaded or clicked something that allowed North Korea to compromise the system.

Then I looked back at my blog. Eight days before the attack, I had blogged about C2PA and used AI-generated pictures of North Korea's leader, Kim Jong Un, as my example. Here's the opening of the blog:

There's nothing worse than a depressed, drunk man who has his finger on the nuclear button.

It appears that this was enough to upset the North Korean government and make me a target for a massive network attack.

Hiding For Safety

Since I'm not taking down my blog, I decided to take additional steps in case the DDoS started up again.

There are some online services that provide DDoS protection. I looked into them and decided to switch to CloudFlare. What they provide:

• Domain fronting. When you connect to hackerfactor.com or fotoforensics.com, you actually connect to one of CloudFlare's servers. They forward the request back to my services. If there is a network attack, then it will hit CloudFlare and not me.


• DDoS protection. I kind of felt bad for setting up CloudFlare for an attack. However, this is one of the things they explicitly offer: DDoS protection, even at the free account level.


• Content caching. By default, they will cache web content. This way, if a hundred people all ask for my blog, I only have to provide it once to CloudFlare. This cuts down on the network volume.


• Filtering rules. Even at the free tier, you can create filtering rules to stop bots, AI-scrapers, block bullet-proof hosting providers, etc. (I'm using their paid tier for some of my domains because I wanted more filter options.)

Setting up an account and moving my main domains took hours -- not days or months.

The downside of using CloudFlare is that I like to monitor my network attacks. Since CloudFlare gets these attacks instead of me, I don't have that insight. However, I still run some honeypots outside of CloudFlare so I still have baseline attack metrics.

The Second Wave

Even though my servers had been hit by a massive attack, I decided to slowly move them to the new service. (I'd rather be slow and cautious and get everything right, than to rush it and make a different problem.)

On 28-Oct-2024 (five days after the first attack) at almost exactly 1:00 AM, the attack started again. Although I had moved my servers behind CloudFlare, they appeared to be directly attacking my previously-known location.

Unfortunately, they guessed correctly. Even though CloudFlare was protecting me from incoming attacks, CloudFlare was forwarding valid requests back to my servers. And my servers were still at the old IP addresses. By attacking the old addresses, the DDoS managed to take down my service again.

I called my ISP's emergency 24/7 support number to report the problem, but nobody answered so I left a message. I repeatedly called back every 30-60 minutes until I was able to reach a person -- at 7:20am. (I spoke to the head of my ISP's IT department. They will make sure the 24/7 support will actually be manned next time.) They issued another IP address black hole to stop the attack, and it stopped 20 minutes later.

At this point, I decided to switch around network addresses and bridge in a second ISP. If one ISP goes down, the other one should kick in.

The Third Wave

On 30-Oct-2024, the third wave happened. This one was kind of funny. While my servers were dual homed and on different IP addresses, I still had some equipment using the old addresses. I was working late at night and heard the server fans start up again...

It took me a moment to check all of my diagnostics and determine that, yes, it was the DDoS again. It only took a minute for me to look up the ISP's 24/7 support number. However, as I picked up the phone, I heard the fans rev down. (Odd.) A few seconds later, a different server began revving up. After a minute, it spun down and a third server revved up.

That's when I realized what the attacker was doing. I had a sequential block of IP addresses. They were DDoS'ing one address and checking if my server went offline. After a minute, they moved the DDoS to the next IP address, then the next one. Here's the problems they were facing:

• I had moved my main services to different addresses. This meant that the attacker couldn't find me.


• My services were behind CloudFlare and they cache content. Even if the attacker did find me, their polling to see if I was down would see cached content and think I was still up.

Later that day, CloudFlare posted about a massive DDoS that they had prevented.

Cloudflare
@cloudflare@noc.social

We recently thwarted a massive UDP Flood attack from 8-9K IPs targeting ~50 IP addresses of a Magic Transit customer. This was part of a larger campaign we covered in our Q3 2024 report. Check out the full details here: https://blog.cloudflare.com/ddos-threa...

5.6 terabits per second. Wow. When I wrote to CloudFlare asking if this was related to me, I received no reply. I'm certainly not saying that "this was due to me", but I kind of suspect that this might have been due to me. (Huge thanks to CloudFlare for offering free DDoS protection!)

Keep in mind, CloudFlare says that they can handle 296 terabits per second, so 5.4Tbps isn't going to negatively impact them. But I can totally understand why my (now former) ISP couldn't handle the volume.

Tricks, Treats, and Terabits

I did lay out a couple of detectors and devised a few ways to automatically redirect this attack toward other targets. However, it hasn't resurfaced in a year. (I really wanted to redirect North Korea's high-volume DDoS attack against Russian targets. Now that I've had time to prepare a proper response, I'm sure I can do the redirection with no impact to my local network. I mean, they watch my DNS, so I'd just need to change my DNS to point to Russia. I wonder if this redirected attack would cause an international incident?)

Halloween stories usually end when the monster is vanquished. The lights come back on, the hero breathes a sigh of relief. But for system administrators, the monsters don't die; they adapt. They change IPs, morph signatures, and wait for a moment of weakness.

Some people fear ghosts or ghouls. I fear the faint whine of server fans spinning up in the middle of the night. A sound that means something, somewhere, has found me again. The next time the servers ramps up, it might not be an innocent workload. It might be the North Korea bot army.

Everyone with a new project or new technology wants rapid adoption. The bigger the customer base, the more successful the project can hope to be. However, in the race to be the first one out the door, these developers often overlook the customer needs. (Just because you can do it, does it mean the customer wants it? And does it solve a problem that the customer has?)

Many of my friends, colleagues, and clients work in security-related industries. That explicitly means that they are risk-averse and very slow to adopt new technologies. Even for existing technologies, some of my clients have multi-month release cycles. Between when I give them a code drop and when they deploy it for their own use, a few months might pass. During that time, the code is scanned for malware, tested in an isolated testing environment (regression testing), then tested in a second testing environment (more aggressive testing), maybe a third testing environment (real-world simulation), and then finally deployed to production. It's rare for large companies to just download the code drop and start using it in production.

There's a big reason for being risk-averse. If you work in the financial, medical, or insurance fields, then you have legal liability. You're not going to adopt something that puts you or your company at risk. If you can't trust that a tool's results will be consistent or trustworthy, then you're not going to use it.

In this blog, I explore whether C2PA-signed media, such as photos from the Google Pixel 10, should be accepted as reliable evidence in a court of law. My findings show serious inconsistencies in timestamps, metadata protection, and AI processing. These issues call its forensic reliability into question.

Legally Acceptable

Forensics effectively means "for use in a court of law". For any type of analysis tool, the biggest concern is whether the tool or the results are admissible in court. This means that it needs to comply with the Daubert or Frye standards and the Federal Rules of Evidence. These are the primary requirements needed to ensure that tools and their results can be admissible in a US courtroom.

Daubert and Frye refer to two different guidelines for accepting any tools in a court of law. As a non-attorney, my non-legal understanding is that they are mostly the same thing. Both require the tools, techniques, and methods to be relevant to the case, scientifically sound, and provide reliable interpretations. The main differences:

• Daubert is used in federal and some state courts. Frye is used in the states that don't rely on Daubert. However, you might hear arguments related to both the Daubert and Frye interpretations in a single courtroom.


• Frye is based on a "general acceptance" criteria: Is the underlying scientific principle or method "generally accepted" as reliable within the relevant scientific community?


• Daubert requires the judge to act as the final "gatekeeper". The judge uses specific criteria to evaluate the principles and methodology (not the conclusion generated) before determining if the approach is acceptable. Because Daubert considers multiple factors (such as the error rate and methods used), it is often considered to be a stricter standard.


• In both cases, judges often rely on precedence. If your tool is accepted in court one time, then it's more likely to be accepted the next time.

Along with Daubert and Frye, the Federal Rules of Evidence (FRE) define criteria for acceptability of both the evidence and anyone testifying about the evidence. These include guidance about relevancy (FRE Rules 401 and 403), expert testimony (FRE Rule 702), and scientific acceptability (FRE Rules 901 and 902). I'm not an attorney, and I'm sure that legal experts can identify additional requirements.

For my FotoForensics service:

• All analyzers are based on solid (logical) theory. The outputs are deterministic and repeatable. Moreover, other people have been able to implement variations of my analyzers and can generate similar results. (This goes toward reproducibility.)


• I explicitly avoid using any kind of deep learning AI. (I don't use AI to detect AI, which is the current fad.) This goes toward provability. In fact, every outcome on the commercial FotoForensics service is easily explainable -- it's a white box system, not a black box. When the expert gets up on the stand, they can explain how the software reached any results.


• FotoForensics has been peer reviewed and referenced in academic publications. The back-end analysis tools also passed a functional review by the Department of Defense Cyber Crime Center (DC3). (The DC3 only gives a pass/fail rating. It passed, with the note "steep learning curve". FotoForensics was created as a front-end to simplify the learning curve.)


• FotoForensics has been used in a court of law and deemed acceptable. (Technically, the tool itself was never questioned. The experts using the tool were found to be acceptable under FRE Rule 702.)

In contrast to FotoForensics, we have cameras adopting AI content creation and integrating with C2PA. With the Google Pixel 10, we have them combined. Each poses a problem and when combined, they make the problems worse.

Vetting the Pixel 10

Let's say you have a picture from Google's new Pixel 10 smartphone and you want to use it as evidence in a court of law. Is the picture reliable? Can the C2PA signature be used to authenticate the content?

Keep in mind, if you are law enforcement then you are a trained observer and sworn to uphold the law. In that case, saying "Yes, that picture represents the situation that I observed" is good enough. But if you're anyone else, then we need to do a deep analysis of the image -- in case you are trying to submit false media as evidence. This is where we run into problems.

To illustrate these issues, I'm going to use three pictures that "named router" (that's what he calls himself) captured. He literally walked in to a store, asked to test a Pixel 10 camera, and took three photos in succession. Click click click. (He gave me permission to use them in this blog.) Here are the pictures with links to the analyzers at FotoForensics, Hintfo, and C2PA's official Content Credentials website, as well as the EXIF timestamp and the trusted timestamp (TsT) from the notary that are found in the picture's metadata:

Image	Analyzers	EXIF vs TsT Difference	Metadata Consistency
	FotoForensics Hintfo Content Credentials	EXIF: 2025-10-14 17:19:25 +02:00 Notarized: 2025-10-14 15:19:26 GMT Difference: +1 second	Consistent metadata and expected behavior.
	FotoForensics Hintfo Content Credentials	EXIF: 2025-10-14 17:19:32 +02:00 Notarized: 2025-10-14 15:19:32 GMT Difference: 0 seconds	Includes a video; an unexplained variation.
	FotoForensics Hintfo Content Credentials	EXIF: 2025-10-14 17:19:36 +02:00 Notarized: 2025-10-14 15:19:35 GMT Difference: -1 seconds	Metadata is consistent with the first image. However, the timestamps are inconsistent: the notary predates the EXIF.

If we accept the EXIF and XMP metadata at face value, then:

• The pictures were taken seconds apart. (Three pictures in 11 seconds.)


• The EXIF metadata identifies each picture as coming from a Google Pixel 10 camera. Each includes lens settings and device information that seem rational. The EXIF data matches the expectations.


• There is one problem in the EXIF metadata: Every picture created by the Pixel 10 has EXIF metadata saying that it is a composite image: Composite Image Captured While Shooting. (Technically, this is EXIF tag 0xa460 with value 3. "3" means it is a composite image that was captured while shooting.) This tag means that every picture was adjusted by the camera. Moreover, it was more than the usual auto-exposure and auto-focus that other cameras provide. We do not know if the alterations significantly altered the meaning of the picture. Given that every picture from a Pixel 10 includes this label, it's expected but problematic.


• Comparing the files, there is a bigger problem. These files were taken in rapid succession. That means the user didn't have time to intentionally change any camera settings. (I confirmed this with the photographer.) The first and third pictures have the same metadata fields in the same order, and both include a Gain Map. (Consistency is good.) However, the second picture includes an additional video attachment. Inconsistency is bad because it can appear to be tampering. (Talking with the photographer, there was nothing special done. We have no idea why the camera decided to randomly attach a video to the JPEG image.)

Many cameras change metadata based on orientation, zoom, or other settings. However, I don't know of any other cameras that arbitrarily change metadata for no reason.

Metadata, by itself, can be maliciously altered, but that usually isn't the default assumption. Instead, the typical heuristic is: trust the metadata until you seen an indication of alteration. At the first sign of any inconsistency, stop trusting the metadata. The same goes for the content: trust the content until you notice one inconsistency. The instant you notice something wrong, you need to call everything into question.

Add in C2PA

Given that every Pixel 10 picture natively includes AI and can randomly change the metadata, this can be very problematic for an examiner. However, it gets worse, because the Pixel 10 includes C2PA metadata.

As mentioned in my earlier blog entry, the Pixel 10's C2PA metadata (manifest) explicitly excludes the EXIF and XMP metadata from the cryptographic signature. Those values can be easily changed without invalidating the cryptographic signature. Again, this doesn't mean that the metadata was altered, but it does mean that the metadata could be altered without detection.

The C2PA metadata includes a timestamp from a trusted source. This is not when the picture was generated. This is when the picture was notarized. It says that the trusted source saw that the data existed at the time it was signed.

What I didn't know at the time I wrote my previous blog entry was that the Pixel 10's Trusted Timestamp Authority (TSA) is built into the camera! They appear to maintain two separate clocks in the Pixel 10. One clock is for the untrusted EXIF timestamps (when the file was created), while the other is for the trusted timestamp (when the notary signed the file). This is where the problem comes in: if both clocks exist on the same device, then I'd expect drift between the clocks to be negligible and times to be consistent.

• In the first sample picture, the notary's timestamp is one second after the EXIF timestamp. This doesn't bother me since the actual difference could be a fraction of a second. For example, the EXIF data says the picture was captured at 17:19:25.659 seconds (in GMT+02:00). If the signing took 0.35 seconds, then the trusted timestamp would be at 26.xxx seconds -- with integer rounding, this would be 15:19:26 GMT, which matches the timestamp from the trusted notary.


• The second picture was captured at 17:19:32 +02:00 and notarized within the same second: 15:19:32 GMT. This is consistent.


• The third picture was captured at 17:19:36 +02:00 and notarized one second earlier! At 15:19:35 GMT. This is a significant inconsistency. Since it's all on the same device, the notary should be at or after the EXIF is generated; never before. Moreover, since the trusted timestamp shows that the file was notarized before the EXIF data was generated, we don't know what information was actually notarized.

Without C2PA, I would have had no reason to distrust the EXIF timestamps. But with C2PA, I now have a strong reason to suspect that a timestamp was altered. Moreover, we have two inconsistencies among these three pictures: one with different metadata and one with the notary generating a signed timestamp before the data was created.

As an analyst with no information about the ground truth, it's not just that I wouldn't trust the third picture; I wouldn't trust the entire set! It looks no different than if someone modified the EXIF timestamps on the third picture, and likely altered the first two pictures since the alteration would be undetectable. Moreover, the metadata difference in the second picture could be from someone inserting a different image into the sequence. (That's common with forgeries.)

Unfortunately, knowing the ground truth doesn't make this any better.

With other Google Pixel 10 images, I've seen the unaltered EXIF times and trusted timestamp differ by almost 10 seconds. For example, this haystack image came from DP Review:



The metadata says that the photo was captured on 2025-08-27 01:02:43 GMT and notarized a few seconds later, at 2025-08-27 01:02:46 GMT. The problem is, there's also a depth map attached to the image, and it also contains EXIF metadata. The main image's EXIF data is part of the C2PA exclusion list, but the depth map (and its EXIF data) is protected by a C2PA's cryptographic signature. The depth map's EXIF says it was created at "2025-08-27 01:02:47 GMT", which is one second after the picture was notarized.

• The entire picture took four seconds to be created.
• The notary signed the trusted timestamp before the camera finished creating the file.

This doesn't just call into question the trusted timestamp in the initial three pictures or the haystack photo. This calls into question every trusted timestamp from every C2PA-enabled application and device.

Time's Up

Back in 2024, I gave a presentation for the IPTC organization titled "C2PA from the Attacker's Perspective". During the talk, I demonstrated some of the problems that I had previously (privately) reported to the C2PA. Specifically, I showed how anyone could alter the trusted timestamp without detection. (Some of C2PA's steering committee members had previously heard me say it was possible, but they had not seen a working demonstration of the exploit.) There were two core problems that made this alteration possible:

1. The C2PA implementation by Adobe/CAI was never checking if the trusted timestamp was altered. This meant that I could alter it without detection. (It was pretty entertaining when I demonstrated this live, and people were commenting in the real-time chat with remarks like "I just confirmed it.")


2. As far as I could tell, the trusted timestamp was not protecting anything else. The trusted timestamp is supposed to be cryptographically encompassing some other data:
   
   
   1. You generate a hash of the data you want to sign.
   2. You send the hash to the Trusted Timestamp Authority (TSA).
   3. The TSA returns a certificate with a signed timestamp.
   
   Although my IPTC presentation criticized an earlier version of the C2PA specification, the most recent version (C2PA v2.2) retains the same problem. Unfortunately, C2PA's implementation doesn't appear to signing anything of importance. This appears to be a problem that stems from the C2PA specifications. Specifically, section 10.3.2.5 mentions signing a minor portion of the manifest: (my bold for emphasis)
   

   10.3.2.5.2. Choosing the Payload
   
   A previous version of this specification used the same value for the payload field in the time-stamp as was used in the Sig_signature as described in Section 10.3.2.4, “Signing a Claim”. This payload is henceforth referred to as a "v1 payload" in a "v1 time-stamp" and is considered deprecated. A claim generator shall not create one, but a validator shall process one if present.
   
   The "v2 payload", of the "v2 time-stamp", is the value of the signature field of the COSE_Sign1_Tagged structure created as part of Section 10.3.2.4, “Signing a Claim”. A "v2 payload" shall be used by claim generators performing a time-stamping operation.
   
   
   10.3.2.4. Signing a Claim
   
   Producing the signature is specified in Section 13.2, “Digital Signatures”.
   
   For both types of manifests, standard and update, the payload field of Sig_structure shall be the serialized CBOR of the claim document, and shall use detached content mode.
   
   The serialized COSE_Sign1_Tagged structure resulting from the digital signature procedure is written into the C2PA Claim Signature box.
   
   ...
   14.2.2. Use of COSE
   
   Payloads can either be present inside a COSE signature, or transported separately ("detached content" as described in RFC 8152 section 4.1). In "detached content" mode, the signed data is stored externally to the COSE_Sign1_Tagged structure, and the payload field of the COSE_Sign1_Tagged structure is always nil.

   It appears that other parts of the manifest (outside of the "v2 payload") can be altered after generating the trusted timestamp's signature. This gives the appearance of claiming that the timestamp authenticates the content or manifest, even though it permits the manifest to be changed after generating the trusted timestamp information.

After my live demo, the Adobe/CAI developers fixed the first bug. You can no longer manually alter the trusted timestamp without it being flagged. If you use the command-line c2patool, any attempt to alter the trusted timestamp returns a "timeStamp.mismatch", like:

"informational": [

  {

    "code": "timeStamp.mismatch",

    "url": "self#jumbf=/c2pa/urn:c2pa:2c33ba25-6343-d662-8a8c-4c638f4a3e68",

    "explanation": "timestamp did not match signed data"

  }

However, as far as I can tell, they never fixed the second problem: the trusted timestamp isn't protecting anything important. (Well, other than itself.). The manifest can be changed after the trusted timestamp is generated, as long as the change doesn't touch the few bytes that the C2PA specifications says to use with the trusted timestamp.

With C2PA, the trusted timestamp is supposed to work as a notary. However, as a notary, this is effectively signing a blank piece of paper and letting someone fill in everything around it. The trusted timestamp appears to be generated before the file is fully created and the exclusion list permits the EXIF and XMP metadata to be altered without detection. In any legal context, these fatal flaws would immediately void the notary's certification.

This isn't just a problem with every Google Pixel photo. This seems to be a problem with every signed C2PA image that includes a TSA's trusted timestamp. Since the Google Pixel 10 is on C2PA's conforming product list, this is a problem with the C2PA specification and not the individual implementations.

Core Arguments

At this point, we have a photo from a Pixel 10 with a C2PA signature. What can we trust from it?

• The Pixel 10 uses AI to alter the image prior to the initial saving. As demonstrated in the previous blog entry, the alterations are extreme enough to be confused with AI-generated images. There appears to be no way to disable these alterations. As a result of the AI manipulation, it becomes questionable whether the alterations change the media's meaning and intent.


• The Pixel 10 does not protect the EXIF or XMP metadata from tampering. (Those metadata blocks are in the C2PA manifest's exclusion list.) An analyst should completely ignore the C2PA metadata and evaluate the EXIF and XMP independently.


• The trusted timestamp appears to be mostly independent of the picture's creation time. (I say "mostly" because the camera app appears to call the TSA "sometime" around when the picture's C2PA manifest was being created.)

If the C2PA cryptographic signatures are valid, it only means that the file (the parts that were not excluded from the signature) was not altered after signing; it says nothing about the validity of the content or accuracy of the metadata. A valid C2PA manifest can be trusted to mean that parts of the file are cryptographically unaltered since the camera signed it. However, it does not mean that the visual content is a truthful representation of reality (due to AI) or that the metadata is pristine (due to exclusions).

But what if the C2PA signatures are invalid, or says that the media was altered? Does that mean you can still trust it? Unfortunately, the answer is "no". Just consider the case of the Holocaust deniers. (I blogged about them back in 2016.) Some Holocaust deniers were manufacturing fake Holocaust pictures and getting them inserted into otherwise-legitimate photo collections. Later, they would point out their forgeries (without admitting that they created them) in order to call the entire collection into question.

By the same means, if someone attaches an invalid C2PA manifest to an otherwise-legitimate picture, it will appear to be tampered with. If people believe that C2PA works, then they will assume that the visual content was altered even through the alteration involved the manifest. (This is the "Liar's Dividend" effect.)

Regardless of whether the C2PA's cryptographic signatures are valid or invalid, the results from the C2PA manifest cannot be trusted.

Summary Judgment

All of this takes us back to whether pictures from a Google Pixel 10, and C2PA in general, should be accepted for use in a court of law. (To reiterate, I'm not a legal scholar and this reflects my non-attorney understanding of how evidence and forensic tools are supposed to be presented in court.)

• FRE Rule 901(b)(9) covers "Evidence About a Process or System". The proponent must show the "process or system produces an accurate result." In this blog entry, I have shown that the "Pixel 10/C2PA system" fails this on several counts:
  
  
  • Lack of "Accurate Result": The AI/Computational Photography from the Pixel 10 means the image is not a simple, accurate record of light, but an interpretation.
  
  
  • System Inconsistency: The random inclusion of a video attachment and the random, unexplained time shifts demonstrate that the "system" (the camera software) is not operating consistently or predictably, thus failing to show that it "produces an accurate result" in a reliably repeatable manner.
  
  
  • Lack of Relevancy: The trusted timestamp does not appear to protect anything of importance. The manifest and contents can be written after the Trusted Timestamp Authority generates a signed timestamp.


• FRE Rule 902 goes toward self-authentication. It is my understanding that this rule exists to reduce the burden of authentication for certain reliable records (like certified business records or electronic process results). By failing to cryptographically protect the foundational metadata (EXIF/XMP), the C2PA system prevents the record from even being considered for self-authentication under modern amendments like FRE Rule 902(13) or 902(14) (certified electronic records and data copies), forcing it back to the more difficult FRE Rule 901 requirements.


• FRE Rule 702 (Expert Testimony) and black boxes. FRE Rule 702 requires an expert's testimony to be based on "reliable principles and methods" which the expert has "reliably applied to the facts of the case." With AI-driven "computational capture", the expert cannot explain why the image looks the way it does, because the specific AI algorithm used to generate the final image is a proprietary black box. This is a significant Daubert and FRE Rule 702 hurdle that the Pixel 10 creates for any expert trying to authenticate the image.


• The Kuhmo Tire, Joiner and Rule 702 Precedent. Even if the C2PA specification is found to be generally valid (Daubert/Frye), the Google Pixel 10's specific implementation may fail the Kuhmo Tire test because of its unreliable application (the timestamp reversal, the random video attachment, and the exclusion list). An attorney could argue that the manufacturer's implementation is a sloppy or unpredictable application of the C2PA method.

These failures mean that any evidence created by this system raises fundamental challenges. It fails the reliability, reproducibility, and scientific soundness of any media created by the Pixel 10 and of C2PA in general. Based on these inconsistencies, I believe it would likely fail the Daubert, Frye, and Rule 901 standards for admissibility.

The flaws that I've shown, including inconsistent timestamps and partial signing, are not edge cases or vendor oversights. They are baked into the specification's logic. Even a flawless implementation of C2PA version 2.2 would still produce unverifiable provenance because the standard itself allows essential data to remain outside the trusted boundaries.

Good intent doesn't make a flawed protocol trustworthy. In forensic work, evidence is either verifiable or it isn't. If a standard cannot guarantee integrity, then it doesn't matter how many companies endorse it since it fails to be trustworthy.

On Tuesday Oct 21 (tomorrow), the C2PA/CAI is holding a talk titled, "Beyond a Reasonable Doubt: Authenticity in Courtroom Evidence". (Here's the announcement link that includes the invite.) I've tried to reach out to both the speaker and Adobe about this, but nobody responded to me. I certainly hope people go and ask the hard questions about whether C2PA, in its current form, should be admissible in a court of law. (It would be great if some knowledgeable attorneys showed up and asked questions related to Daubert, Frye, Kuhmo Tire, Joiner, Rule 901, Rule 902, Rule 702, etc.) My concern is that having an Assistant District Attorney present in this forum may give false credibility toward C2PA. While I have no reason to doubt that the Assistant District Attorney is a credible source, I believe that the technology being reviewed and endorsed by association is not credible.

Last month, one of my PTSD (People Tech Support Duties) requests led me down a deep path related to AI-alterations inside images. It began with a plea to help photograph a minor skin irritation. But this merged with another request concerning automated AI alterations, provenance, and detection. Honestly, it looks like this rush to embrace "AI in everything" is resulting in some really bad manufacturer decisions.

What started with a request to photograph a minor skin rash ended up spiraling into a month-long investigation into how AI quietly rewrites what we see.

Cameras Causing Minor Irritations

The initial query came from a friend. Her kid had a recurring rash on one arm. These days, doctor visits are cheaper and significantly faster when done online or even asynchronously over email. In this case, the doctor sent a private message over the hospital's online system. He wanted a photo of the rash.

This sounds simple enough. Take out the camera, hold out the arm, take a photo, and then upload it to the doctor. Right?

Here's the problem: Her camera kept automatically applying filters to make the picture look better. Visually, the arm clearly had a rash. But through the camera, the picture just showed regular skin. It was like one of those haunted house scenes where the mirror shows something different. The camera wasn't capturing reality.



These days, smart cameras often automatically soften wrinkles and remove skin blemishes -- because who wants a picture of a smiling face with wrinkles and acne? But in this case, she really did want a photo showing the skin blemishes. No matter what she did, her camera wouldn't capture the rash. Keep in mind, to a human seeing it in real life, it was obvious: a red and pink spotted rash over light skin tone. We tried a couple of things:

• Turn off all filters. (There are some hidden menus on both Android and iOS devices that can enable filters.) On the Android, we selected the "Original" filter option. (Some Androids call this "None".) Nope, it was still smoothing the skin and automatically removing the rash.


• Try different orientations. On some devices (both Android and iOS), landscape and portrait modes apply different filters. Nope, the problem was still present.


• Try different lighting. While bright daylight bulbs (4500K) helped a little, the camera was still mitigating most of it.


• Try a different camera. My friend had both an Android phone and an Apple tablet; neither was more than 3 years old. Both were doing similar filterings.

We finally did find a few ways to get good pictures of the rash:

• Use a really old digital camera. We had a 10+ year old Sony camera (not a phone; a real standalone camera). With new batteries, we could photograph the rash.


• On my older iPhone 12 mini, I was able to increase the exposure to force the rash's red tint to stand out. I also needed bright lighting to make this work. While the colors were far from natural, they did allow the doctor to see the rash's pattern and color differential.


• My laptop has a built-in camera that has almost no intelligence. (After peeling off the tape that I used to cover the camera...) We tried a picture and it worked well. Almost any desktop computer's standalone webcam, where all enhancements are expected to be performed by the application, should be able to take an unaltered image.

I'm glad my friend's kid found this entire experimentation process fascinating. But if this had been a more time-sensitive issue, I honestly don't know what a typical user with a newer device could have done.

This irritating experience was just a scratch of a much larger issue that kept recurring over the month. Specifically, how modern cameras' AI processing is quietly rewriting reality.

AI Photos

Since the start of digital photography, nearly all cameras have included some form of algorithmic automation. Normally it is something minor, like auto-focus or auto-contrast. We usually don't think of these as being "AI", but they are definitely a type of AI. However, it wasn't until 2021 when the first camera-enabled devices with smart-erase became available. (The Google Pixel 6, Samsung Galaxy S21, and a few others. Apple didn't introduce its "Clean Up" smart erase feature until 2024.)

Following the rash problem, I had multiple customer requests asking whether their pictures were real or AI. Each case concerned the same camera: The new Google Pixel 10. This is the problem that I predicted at the beginning of last month. Specifically, every picture from the new Google Pixel 10 is tagged by Google as being processed by AI. This is not something that can be turned off. Even if you do nothing more than bring up the camera app and take a photo, the picture is tagged with the label:

Digital Source Type: http://cv.iptc.org/newscodes/digitalsourcetype/computationalCapture

According to IPTC, this means:

The media is the result of capturing multiple frames from a real-life source using a digital camera or digital recording device, then automatically merging them into a single frame using digital signal processing techniques and/or non-generative AI. Includes High Dynamic Range (HDR) processing common in smartphone camera apps.

In other words, this is a composite image. And while it may not be created using a generative AI system ("and/or"), it was definitely combined using some kind of AI-based system.

In industries that are sensitive to fraud, including banking, insurance, know-your-customer (KYC), fact checking, legal evidence, and photojournalism, seeing any kind of media that is explicitly labeled as using AI is an immediate red flag. What's worse is that analysis tools that are designed to detect AI alterations, including my tools and products from other developers, are flagging Pixel 10 photos as being AI. Keep in mind: Google isn't lying -- every image is modified using AI and is properly labeled. The problem is that you can't turn it off.

One picture (that I'm not allowed to share) was part of an insurance claim. If taken at face value, it looked like the person's car had gone from 60-to-zero in 0.5 seconds (but the tree only sustained minor injuries). However, the backstory was suspicious and the photos, from a Google Pixel 10, had inconsistencies. Adding to these problems, the pictures were being flagged as being partially or entirely AI-generated.

We can see this same problem with a sample "original" Pixel 10 image that I previously used.



At FotoForensics, the Error Level Analysis (ELA) permits visualizing compression artifacts. All edges should look similar to other edges, surfaces should look like surfaces, and similar textures should look similar. With this image, we can see a horizontal split in the background, where the upper third of the picture is mostly black, while the lower two thirds shows a dark bluish tinge. The blue is due to a chrominance separation, which is usually associated with alterations. Visually, the background looks the same above and below (it's the same colors above and below), so there should not be a compression difference. The unexpected compression difference denotes an alteration.



The public FotoForensics service has limited analyzers. The commercial version also detects:

• A halo around the light fixture, indicating that either the background was softened or the chandelier was added or altered. (Or all of the above.)


• The chevrons in the stained glass were digitally altered. (The Pixel 10 boosted the colors.)


• The chandelier has very strong artifacts that are associated with content from deep-learning AI systems.

None of these were intentional alterations. (Jeff just opened the camera app and took a picture. Nothing fancy by the human.) These are all AI-alterations by the Google Pixel 10 and they cannot be disabled.

In my previous blog entry, I showed that Google labels all photos as AI and that the metadata can be altered without detection. But with these automatic alterations baked into the image, we can no longer distinguish reality from revision.

Were the pictures real? With the car photos (that I cannot include here), my professional opinion was that, ignoring the AI and visual content, the photos were being misrepresented. (But doesn't the Pixel 10 use C2PA and sign every photo? Yes it does, but it doesn't help here because the C2PA signatures don't protect the metadata.) If I ignored the metadata, I'd see the alterations and AI fingerprints, and I'd be hard-pressed to determine if the detected artifacts were human initiated (intentional) or automated (unintentional). This isn't the desired AI promise, where AI generates content that looks like it came from a human. This is the opposite: AI forcing content from a human to look like AI.

Other Tools

After examining how these AI-enabled systems alter photos, the next question becomes: how well can our current tools even recognize these changes?

My analysis tools rely on deterministic algorithms. (That's why I call the service "FotoForensics" -- "Forensics" as in, evidence suitable for a court of law.) However, there are other online services that use AI to detect AI. Keep in mind, we don't know how well these AI systems were trained, what they actually learned, what biases they have, etc. This evaluation is not a recommendation to use any of these tools.

This inconsistency between different AI-based detection tools is one of the big reasons I don't view any of them as serious analyzers. For the Pixel 10 images, my clients had tried some of these systems and saw conflicting results. For example, using the same "original" Pixel 10 baseline image:

• Hive Moderation trained their system to detect a wide range of specific AI systems. They claim a 0% chance that this Pixel 10 photo contains AI, because it doesn't look like any of the systems they had trained on. Since the Pixel 10 uses a different AI system, they didn't detect it.
  
  


• Undetectable AI gives no information about what they detect. They claim this picture is "99% REAL". (Does that mean it's 1% fake?)
  
  


• SightEngine decided that it was "3%" AI, with a little generative AI detected.
  
  


• Illuminarty determined that it was "14.9%" AI-generated. I don't know if that refers to 14.9% of the image, or if that is the overall confidence level.
  
  


• At the other extreme, Was It AI determined that this Google Pixel 10 picture was definitely AI. It concluded: "We are quite confident that this image, or significant part of it, was created by AI."
  
  

The ground truth is that the Pixel 10 always uses AI to auto-enhance the picture. If you work in a field that forbids any AI enhancement, then the Pixel 10 is a serious problem. (You can't just tell your client that they need to go back to the site of the accident and take pictures with a different camera.)

Fear the Future

Once upon a time, "taking a picture" meant pressing a button and capturing something that looked like reality. Today, it's more like negotiating with an algorithm about what version of reality it's willing to show you. The irony is that the more "intelligent" cameras become, the less their output can be trusted. When even a simple snapshot passes through layers of algorithmic enhancement, metadata rewriting, and AI tagging, the concept of an "original" photo starts to vanish.

People use AI for lots of tasks these days. This includes helping with research, editing text, or even assisting with diagnostics. However, each of these uses still leaves the human with the final decision about what to accept, reject, or cross-validate. In contrast, the human photographer has no option to reject the AI's alterations to these digital photos.

From medical photos and insurance claims to legal evidence, the line between "photo" and "AI-enhanced composite" has blurred. For fields that rely on authenticity, that's not a minor inconvenience; it's a systemic problem. Until manufacturers return real control to the photographer, sometimes the most reliable camera is the old one in the junk drawer -- like a decade-old Sony camera with no Wi-Fi, no filters, and no agenda.

P.S. Brain Dead Frogs turned this blog entry in a song for an upcoming album. Enjoy!

What do you do when you find a flaw in a piece of computer software or hardware? Depending on the bug, a legitimate researcher might:

• Report it to the vendor. This is the most desirable solution. It should be easy to find a contact point and report the problem.


• Publicly tell others. Full disclosure and public disclosure, especially with a history showing that you already tried to contact the vendor, helps everyone. Even if there is no patch currently available, it still helps other people know about the problem and work on mitigation options. (Even if you can't patch the system, you may be able to restrict how the vulnerable part is accessed.)


• Privately tell contacts. This keeps a new vulnerability from being exploited publicly. Often, a vendor may not have a direct method of reporting, but you might know a friend of a friend who can report it to the vendor through other means.


• Privately sell it. Keeping vulnerabilities quiet also permits making money by selling bugs privately to other interested parties. Of course, you don't know how the others are going to use the new exploit... But that's why you should try to report it to the vendor first. If the vendor isn't interested, then all bets are off. (You can get a premium if you can demonstrate a working exploit and show that the vendor is not interested in fixing it.)


• Keep it to yourself. While there is a risk of someone else finding the same problem, sometimes it's better to keep the bug handy in case you need it in the future. (This is especially true if compromising systems is part of your job description, such as professional penetration testers or hired guns.)


• Do nothing. This option is unfortunately common when the reporting method is unidentified or overly complicated. (I'm trying to report a bug, not build a desk from Ikea. I don't want pages of instructions and an Allen wrench.)

Reporting to the vendor, or at least trying to report to the vendor before resorting to some other option, falls under the well-known best practices of responsible disclosure and full disclosure.

Incentives

Some vendors offer incentives in order to receive bugs. These bug bounty programs often include financial rewards in exchange for informing the vendor and working with them to resolve the issue.

In the old days, bug bounties worked pretty well. (I've even made some money that way.) However, over the years, some companies have perverted the incentives. Rather than paying for the information so they can fix it, they pay for the information and require an agreement to legal terms. For example, some companies have attached stipulations to the payments, such as "by agreeing to this transaction, you will not make it public without coordinating the disclosure with us." (And every time you ask, they will say that they have prioritized it or are still working on it, even years later.) More than a few vendors use bug bounties as a way to bury the vulnerability by paying for silence.

I have personally had too many bad experiences with bug bounties and vendors paying for the privilege of being non-responsive. I don't think bounty programs are worth the effort anymore. Additionally, I won't do bug bounty programs if they require enrolling into any service or are associated with some kind of legal agreement. (If they want me to agree to legal terms, then they need to pay for my attorney to review the terms before I sign it.)

In contrast to bounties, some companies send very nice "thank you" response to people who took the effort to report a bug. Often, these are swag and not financial. I've received t-shirts, hats, mugs, stickers, and very nice thank-you letters. Unlike bounties, I've found that each time a vendor sends me an unsolicited "thank you" (even if it's just a nice personalized email), they are responsive and actively fix the bug.

While some people report bugs for the money or swag, I have a different incentive. If I found a bug and it impacts me or my clients, then I want it fixed. The best place to fix it is with the vendor, so I try to report the problem. This is very much a self-serving purpose: I want their code to work better for my needs. Then again, many vendors are incredibly responsive because they want to provide the best solution to their customers. My bug reports help them and me.

Jump Through Hoops

When it comes to bug reporting, the one thing I don't want to do is jump through hoops. The most common hoops include closed lists, scavenger hunts, strict reporting requirements, and forced legal terms.

• Hoop #1: Closed Lists
  Many open source projects want bugs reported to their mailing list, discord channel, IRC server, private forum, or some other service that requires signing up before reporting. For example, Gentoo Linux wants to you sign up with their Bugzilla service in order to submit a bug, and to keep any discussions in their "forums, IRC or mailing lists". Their various discussion lists also require signing up.
  
  For me, this is the same as a vendor who is unreachable. When I want to report a bug, I don't want to join any lists or forums; I just want to report a bug.


• Hoop #2: Scavenger Hunts
  Some organizations have turned "where to report" into a scavenger hunt. The Tor Project used to be a good example of this. Prior to 2019, they wanted you to search through all of their lists and try to find the correct contact point. If you contacted the wrong person, or gave up because you couldn't find the right person, then that's your fault.
  
  However, after years of complaining, the Tor Project finally simplified the reporting process. Now you can easily find multiple ways to report issues to them. (I still think they usually don't do anything when you report to them, but it's a step in the right direction.)


• Hoop #3: Strict Requirements
  Some companies and organizations have strict requirements for reporting. While I'm fine with providing my name and contact information (in case they have questions or need more details), some forms require you to provide the device's serial number, software version, and other information that the reporter may not have.
  
  For example, many years ago I found a problem with a standalone kiosk. (I'm not naming names here.) The vendor only had one way to report problems: use their online form. Unfortunately, the reporting form absolutely would not let me report the problem without providing a valid serial number and software version for the device. The problem is, it's not my kiosk. I don't know the serial number, software version, or even the alphanumeric location code. Moreover, the exploit appeared to work on all of their kiosks. But due to their strict requirements, I was unable to report the problem. (I ended up finding a friend of a friend who had a contact in the company.)
  
  Some software providers only want reports via their Bugzilla service. Bugzilla often has fields for additional information. (Other bug tracking services have similar features.) Unfortunately, I've had some software groups (again, not naming names) eagerly close out the bug report because I didn't have all of their required information. (It's not that I intentionally didn't supply it; I just didn't have that information.) Beyond an automated message telling me that the bug was closed, they never contacted me. As far as I can tell, they never even looked at the bug because the form wasn't complete enough for them. Keep in mind: my bug reports include detailed step-by-step instructions showing how to do the exploit. (In a few of these cases, I ended up selling the exploits privately to interested parties since the vendors were non-responsive.)


• Hoop #4: Mandatory Terms
  Some companies, like Google, don't have a means to report vulnerabilities without agreeing to their terms. Years ago, Google would accept vulnerability reports with no strings attached. But today, they direct everything to their "Bug Hunters" page. Before reporting a bug, you must agree to their terms.
  
  For me, any kind of non-negotiable mandatory agreement is a showstopper. Even though their terms seem simple enough, I cannot agree to them. For example, they expect "ethical conduct". I expect to act ethically, but since it's their terms, they get to determine what is and is not ethical conduct.

I can understand signing up if you want to be paid a bounty. (Google promises some very large payouts for rare conditions.) But often, I'm not looking for a bounty -- I just want to report a bug.

Interestingly, language barriers have never been a problem in my experience. Most companies auto-translate or accept reports in English. This makes the remaining hoops (legal terms, closed lists, etc.) stand out, because they are entirely self-imposed.

The Bug Bounty Scam

As a service provider, I also receive a lot of inquiries from people posing as potential bug reporters. Here's a sample email:

From: EMX Access Control Sensors <no-reply@[redacted]>
To: [redacted]@[redacted].com
Subject: Paid Bug Bounty Program Inquiry enquiry

Name:
Email: yaseen17money@hotmail.com
Message: Hi Team, I’m Ghulam Yaseen, a security researcher. I wanted to ask if you offer a paid bug bounty program or any rewards for responsibly disclosed vulnerabilities. If so, could you please share the details.

Yaseen writes to me often. Each time, from a different (compromised) domain. (I'm not worried about redacting his email since dozens of forums list this entire email in its entirety.) This isn't a real inquiry/enquiry. A legitimate query would come from their own mail server and use their own name as the sender. I think this is nothing more than testing the mail server to see if it's vulnerable.

In contrast, here's another query I received, and it seems to be from real people (names redacted):

Date: Thu, 21 Aug 2025 05:47:47 +0530
From: [Redacted1] <[redated1]@gmail.com>
Cc: [redacted2]@gmail.com, [Redacted1] <redacted1@gmail.com>
Subject: Inquiry Regarding Responsible Disclosure / Bug Bounty Program

Hello Team,

We are [Redacted1] and [Redacted2], independent security researchers specializing in vulnerability discovery and organizational security assessments.

- [Redacted2] – Microsoft MSRC MVR 2023, with prior vulnerability reports to Microsoft, SAP, BBC, and government entities in India and the UK.
- [Redacted1] – Experienced in bug bounty programs, penetration testing, and large-scale reconnaissance, with findings reported to multiple high-profile organizations.

We came across your organization’s security program information and are interested in contributing by identifying and reporting potential vulnerabilities in a responsible manner.Given our combined expertise in deep reconnaissance, application security, and infrastructure assessments, we believe we could contribute to uncovering critical security issues, including hidden or overlooked vulnerabilities.

Before conducting any form of testing, we would like to confirm the following:

1. Does your organization have an active responsible disclosure or bug bounty program?
2. Could you please define the exact scope of assets that are permitted for testing?
3. Are vulnerabilities discovered on assets outside the listed scopes but still belonging to your organization eligible for rewards?
4. Any rules of engagement, limitations, or legal requirements we should be aware of?
5. Bounty reward structure (if applicable).

We follow strict ethical guidelines and ensure all reports include clear technical detail, reproduction steps, and remediation recommendations.

Looking forward to your guidance and confirmation before initiating any further testing.

Best regards,
[Redacted1] & [Redacted2]

There's no way I'm going to respond to them.

• If I'm going to have someone audit my servers, it will be someone I contact and not who contacts me out of the blue. As covered by Krebs's 3 Basic Rules for Online Safety, "If you didn’t go looking for it, don’t install it!" The same applies to unsolicited offers for help. I didn't go looking for this, so I'm certainly not going to allow them to audit my services.


• Responding positively to any of their questions effectively gives them permission to attack your site.

These are not the only examples. I receive this kind of query at least weekly.

• Some of these inquires mention that my server has a known vulnerability. However, they don't want to tell me what it is until after they confirm that I have a bug bounty program. If I don't respond, or respond by saying that I don't offer any bounties, then they never tell me the bug. Assuming they actually found something, then this feels like extortion. (Pay them or they won't tell me.)


• A few people do point out my vulnerabilities. So far, every single case has either been from one of my honeypots or due to my fake /etc/passwd file. (They asked for a password file, so I gave them one. What's the problem?)

The best thing to do if you receive an unsolicited contact like this? Don't respond. Of course, if they do list a vulnerability, then definitely investigate it. Real vulnerabilities should receive real replies.

Sample Pre-reporting Requirements: Google

In my previous blog entry, I wrote about some significant failures in the Google Pixel 10's C2PA implementation. Before writing about it publicly, I tried multiple ways to report the issue:

1. Direct contact: I repeatedly disclosed my concerns about C2PA to a Google representative. Unfortunately, I believe my concerns were disregarded. Eventually the Google contact directed me to report issues through Google's Bug Hunters system. (That's right: the direct contact didn't want to hear about bugs in his own system unless it came through Google's Bug Bounty service.)


2. Google Bug Hunters: Google's Bug Hunters system requires agreeing to Google's vulnerability reporting terms before I could even submit the bug. I wasn't looking for a bounty; I simply wanted to report a serious problem. For me, being forced to accept legal terms before reporting is a showstopper.


3. Private outreach: After I confirmed the flaws in Pixel 10's C2PA functionality, I reached out to my trusted security contacts. In the past, this network has connected me directly to security teams at Amazon, Facebook, and other major vendors. Since Google's C2PA team was non-responsive, I wanted to contact someone in Google's Android security team or legal department; I suspected that they had not independently reviewed the C2PA implementation for its security, trust, and liability implications. (If they had, I doubt this would have shipped in its current form.) Unfortunately, no one had a contact who could receive a report outside of Google's Bug Hunters. (It's really weird how Google directs everything through Bug Hunters.)

At that point, I had exhausted my options. For me, a reporting process that requires accepting legal terms just to submit a vulnerability -- especially when I am already in direct contact with the team responsible -- is a hoop too many. This is why I posted my blog about Pixel 10's C2PA flaws. (Don't be surprised if I eventually post about more Pixel 10's problems without telling Google first. And if someone inside Google reaches out to me, I'd be happy to discuss this directly, without agreeing to any terms.)

Sample Reporting Hoops: Nikon

Around the same time the Pixel 10 was released, Nikon rolled out C2PA support for the Nikon Z6 III camera. Within days, researcher Horshack discovered that he could get any file signed by the camera, which is a serious flaw in the authenticity system. He even released a signed forgery as a demonstration:



If you upload this picture to Adobe's Content Credentials service, it reports that it is a genuine picture from a "Nikon Z6 3" camera, with no indication that it was altered or forged.

To Nikon's credit, the day after DP Review published an article about this, Nikon temporarily suspended their C2PA signing service, saying they had "identified an issue" and would "work diligently" to resolve it. That's a strong response.



Two weeks after disabling the signing service, Nikon announced that they were revoking their signing certificate. (As of this writing, 2025-09-19, the C2PA and CAI have not revoked Nikon's certificate from their list of trusted certificates. Right now, Every Content Credentials service still says the pictures that are signed using a revoked certificate are still valid and trusted.)

It's unclear if Horshack ever tried to report directly to Nikon. When I searched for a security contact point, Nikon only listed HackerOne as their reporting mechanism. HackerOne is a bug bounty system that requires enrollment, personal information, and banking details. If you aren’t seeking a bounty, then this is a major hoop that discourages reporting.

The community response to Horshack's public disclosure was mostly positive, with many people alarmed and grateful that the issue came to light. However, a few commenters criticized the public release, suggesting it might hurt Nikon's reputation. While lawsuits are always a theoretical risk, I would argue that a vendor that only accepts reports through gated programs effectively pushes researchers toward public disclosure as the only viable reporting path.

In this case, Nikon acted quickly once the problem went public. This demonstrates that they can respond, but the process could have been much smoother if they provided a simple, open reporting channel.

When one problem is reported, it's not unusual to see other people identify related problems. In the comments to the original reporting, other people detailed additional issues. For example:

• patrol_taking_9j noted that "NX Studio is completely unable to export JPEG's at all for any RAW or RAW+JPEG NEF files shot with C2PA enabled."


• Horshack replied to his own posting, noting that pictures appear to be signed hours after capture.


• Pierre Lagarde remarked that "The only thing I can say is C2PA still looks like a problem by itself, not that much like a solution to anything. At least, I think the inclusion of this feature at this stage seems premature." (I fully agree.)


• To further demonstrate the problem, Horshack created a second signed forgery:
  
  
  
  As with his first forgery, the Content Credentials service reports that it is a photo from a "Nikon Z6 3" camera.

These additional problems show the power of public disclosure. Had Horshack not made the initial problem public, other people may have not looked as closely and these related concerns may not have been brought to light.

Lower Barriers, Better Security

Bug reporting should not feel like running an obstacle course. Every extra hurdle, whether it's mandatory legal terms, scavenger hunts, closed lists, or bounty-program enrollment, increases the likelihood that a researcher will give up, go public, or sell the exploit privately.

The Google and Nikon cases highlight the same lesson: if you make responsible reporting difficult, then you drive researchers toward public disclosure. That might still result in a fix, but it also increases the window of exposure for everyone who uses the product.

The vendors who handle vulnerability reporting the best are the ones who make it simple: a plain-text email address, a short web form, or even a contact page that doesn't require more than a description and a way to follow up. Many of these vendors don't pay bounties, yet they respond quickly and even say "thank you", which is often enough to keep security researchers engaged.

The industry doesn't need more hurdles. It needs frictionless reporting, fast acknowledgment, and a clear path from discovery to resolution. Good security starts with being willing to listen: make it as easy as possible for the next person who finds a flaw to tell you about it.

Google recently released their latest-greatest Android phone: the Google Pixel 10. The device has been met with mostly-positive reviews, with the main criticisms around the over-abundance of AI in the device.

However, I've been more interested in one specific feature: the built-in support for C2PA's Content Credentials. For the folks who are new to my blog, I've spent years pointing out problem after problem with C2PA's architecture and implementation. Moreover, I've included working demonstrations of these issues; these problems are not theoretical. C2PA is supposed to provide "provenance" and "authenticity" (the P and A in C2PA), but it's really just snake oil. Having a cryptographically verifiable signature doesn't prove anything about whether the file is trustworthy or how it was created.

A Flawed Premise

A great movie script usually results in a great movie, regardless of how bad the actors are. (In my opinion, The Matrix is an incredible movie despite Keanu Reeves' lackluster performance.) In contrast, a bad script will result in a bad movie, regardless of how many exceptional actors appear in the film, like Cloud Atlas or Movie 43. The same observation applies to computer software: a great architecture usually results in a great implementation, regardless of who implements it, while a bad design will result in a bad implementation despite the best developers.

C2PA starts from a bad architecture design: it makes assumptions based on vaporware, depends on hardware that doesn't exist today, and uses the wrong signing technology.

Google Pixel 10

I first heard that the Google Pixel 10 was going to have built-in C2PA support from Google's C2PA Product Lead, Sherif Hanna. As he posted on LinkedIn:

It's official — the Google Pixel 10 is the first smartphone to integrate C2PA Content Credentials in the native Pixel Camera app. This is not just for AI: *every photo* will get Content Credentials at capture, and so will every edit in Google Photos—with or without AI.

Best of all, both Pixel Camera and Google Photos are *conformant Generator Products*, having passed through the C2PA Conformance Program.

If you didn't know better, this sounds like a great announcement! However, when I heard this, I knew it would be bad. But honestly, I didn't expect it to be this bad.

Sample Original Photo

One of my associates (Jeff) received the Google Pixel 10 shortly after it became available. He took a sample photo with C2PA enabled (the default configuration) and sent it to me. Here's the unaltered original picture (click to view it at FotoForensics):



If we evaluate the file:

• Adobe (a C2PA steering committee member) provides the official "Content Credentials" web service for validating C2PA metadata. According to them, all digital signatures are valid. The site reports that this came from the "Google C2PA SDK for Android" and the signature was issued by "Google LLC" on "Aug 28, 2025 at 8:10 PM MDT" (they show the time relative to your own time zone). According to them, the image is legitimate.


• Truepic (another C2PA steering committee member) runs a different "Content Credentials" web service. According to them, "Content credentials are invalid because this file was signed by an untrusted source."
  
  
  
  If we ignore that Truepic haven't updated their trusted certificate list in quite some time, then they claim that the manifest was signed by this signer and that it indicates no AI:
  

  detected_attributes: {
  
      is_ai_generated: false,
  
      is_ai_edited: false,
  
      contains_ai: false,
  
      is_camera_captured: false,
  
      is_visual_edit: false
  
  }

  Both authoritative sites should authenticate the same content the same way. This contradiction will definitely lead to user confusion.


• My FotoForensics and Hintfo services display the metadata inside the file. This picture includes a rich set of EXIF, XMP, and MPF metadata, which is typical for a camera-original photo. The EXIF identifies the make and model (Google Pixel 10 Pro), capture timestamp (2025-08-28 22:10:17), and more. (Jeff didn't include GPS information or anything personal.)


• There's also a C2PA manifest for the "Content Credentials". (It's in the JUMBF metadata block.) FotoForensics shows the basic JUMBF contents, but it's not easy to read. (FotoForensics doesn't try to format the data into something readable because all C2PA information is unreliable. Displaying it will confuse users by giving the C2PA information false credibility.) My Hintfo service shows the parsed data structure:
  
  
  • The manifest says it was created using "Google C2PA SDK for Android" and "Created by Pixel Camera".
  
  
  • There is a cryptographically signed timestamp that says "2025-08-29T02:10:21+00:00". This is not when the picture was created; this is when the file was notarized by Google's online timestamp service. This timestamp is four seconds after the EXIF data says the picture was captured. This is because it required a network request to Google in order to sign the media.
  
  
  • The manifest includes a chain of X.509 certificates for the signing. The signer's name is "Google LLC" and "Pixel Camera". If you trust the name in this certificate, then you can trust the certificate. However, it's just a name. End-users cannot validate that the certificate actually belongs to Google. Moreover, this does not include any unique identifiers for the device or user. Seeing this name is more "branding" than authentication. It's like having "Levi's" stamped on the butt of your jeans.
  
  Notice that the C2PA manifest does not list the camera's make, model, photo capture time, lens settings, or anything else. That information is only found in the EXIF metadata.


• Inside the C2PA actions is a notation about the content:
  

  "digitalSourceType": "http://cv.iptc.org/newscodes/digitalsourcetype/computationalCapture"

  According to IPTC, this means:
  

  The media is the result of capturing multiple frames from a real-life source using a digital camera or digital recording device, then automatically merging them into a single frame using digital signal processing techniques and/or non-generative AI. Includes High Dynamic Range (HDR) processing common in smartphone camera apps.

  
  In other words, this is a composite image. And while it may not be created using a generative AI system ("and/or"), it was definitely combined using some kind of AI-based system.
  
  (Truepic's results are wrong when they say that no AI was used. They are also wrong when they say that it is not from a camera capture. Of course, someone might point out that Truepic only supports C2PA v2.1 and this picture uses C2PA v2.2. However, there is no C2PA version number in the metadata.)
  
  As an aside, Jeff assures me that he just took a photo; he didn't do anything special. But the metadata clearly states that it is a composite: "capturing multiple frames" and "automatically merging them". This same tag is seen with other Pixel 10 pictures. It appears that Google's Pixel 10 is taking the same route as the iPhone: they cannot stop altering your pictures and are incapable of taking an unaltered photo.


• The most disturbing aspect comes from the manifest's exclusion list:
  

  "assertion_store":  {
  
        "c2pa.hash.data":  {
  
          "exclusions":  {
  
            [
  
              {
  
              "start": "6",
  
              "length": "11572"
  
              }
  
            ],
  
            [
  
              {
  
              "start": "11596",
  
              "length": "4924"
  
              }
  
            ],
  
            [
  
              {
  
              "start": "17126",
  
              "length": "1158"
  
              }
  
            ],
  
            [
  
              {
  
              "start": "18288",
  
              "length": "65458"
  
              }
  
            ],
  
            [
  
              {
  
              "start": "83750",
  
              "length": "7742"
  
              }
  
            ]
  
          },

  When computing the digital signature, it explicitly ignores:
  
  
  • 11,572 bytes beginning at byte 6 in the file. That's the EXIF data. None of the EXIF data is protected by this signature. Unfortunately, that's the only part that defines the make, model, settings, and when the photo was taken.
  
  
  • 4,924 bytes starting at position 11,596. That's the JUMBF C2PA manifest. This is the only component that's typically skipped when generating a C2PA record because most of it is protected by different C2PA digital signatures.
  
  
  • 1,158 bytes beginning at position 17,126 is the XMP data.
  
  
  • 65,458 bytes beginning at position 18,288 is the extended XMP metadata that includes Google's Makernotes.
  
  
  • 7,742 bytes beginning at position 83,750 is the continuation of the extended XMP metadata record.
  
  That's right: everything that identifies when, where, and how this image was created is unprotected by the C2PA signature. C2PA's cryptographic signatures only covers the manifest itself and the visual content. It doesn't cover how the content was created.

Without C2PA, anyone can alter the EXIF or XMP metadata. (It's a very common forgery approach.)

With the Google Pixel's C2PA implementation, anyone can still alter the EXIF or XMP metadata. But now there's a digital signature, even if it doesn't identify any alterations.

The problem is that nothing on either of the "Content Credentials" web services reports the exclusion range. If you're a typical user, then you haven't read through the C2PA specifications and will likely assume that the file is trustworthy with tamper-evident protection since the cryptographic signature is valid.

Forgery Time!

Knowing what I can and cannot edit in the file, I altered the image to create a forgery. Here's my forgery:

• If you use the official Adobe/CAI Content Credentials validation tool, you will see that the entire file is still cryptographically sound and shows the same authoritative information. There is no indication of alteration or tampering. (The results at Truepic's validation service also haven't changed.)


• The metadata displayed by FotoForensics and Hintfo shows some of the differences:
  
  
  • The device model is "Pixel 11 Pro" instead of "Pixel 10 Pro". I changed the model number.
  
  
  • The EXIF software version was "HDR+ 1.0.790960477zd". Now it is "HDR+ 3.14156926536zd". (Really, I can change it to anything.)
  
  
  • The EXIF create and modify date has been backdated to 2025-07-20 12:10:17. (One month, 8 days, and 12 hours earlier than the original.)
  
  Although this is all of the EXIF data that I changed for this example, I could literally change everything.


• Hintfo shows the decoded JUMBF data that contains the C2PA manifest. I changed the manifest's UUID from "urn:c2pa:486cba89-a3cc-4076-5d91-4557a68e7347" to "urn:neal:neal-wuz-here-neal-wuz-here-neal-wuz". Although the signatures are supposed protect the manifest, they don't. (This is not the only part of the manifest that can be altered without detection.)

While I cannot change the visual content without generating a new signature, I can change everything in the metadata that describes how the visual content came to exist.

Consistently Inconsistent

Forgeries often stand out due to inconsistencies. However, the Pixel 10's camera has been observed making inconsistent metadata without any malicious intervention. For example:



According to Digital Photography Review, this photo of a truck is an out-of-the-camera original picture from a Pixel 10 using 2x zoom. The EXIF metadata records the subject distance. In this case, the distance claims to be "4,294,967,295 meters", or about 11 times the distance from the Earth to the Moon. (That's one hell of a digital zoom!) Of course, programmers will recognize that as uint32(-1). This shows that the Pixel 10 can naturally record invalid values in the metadata fields.

As another example:



DP Review describes this graffiti picture as another out-of-the-camera original using 2x zoom. It also has the "4,294,967,295 meters" problem, but it also has inconsistent timestamps. Specifically:

• The EXIF metadata has a creation date of "2025-08-25 19:45:28". The time zone is "-06:00", so this is 2025-08-26 01:45:28 GMT.


• The C2PA-compliant external trusted timestamp authority operated by Google says it notarized the file at 2025-08-26 01:45:30 GMT. This means it took about 2 seconds for the signing request to go over the network.


• This picture has a few attached parasites. (A parasite refers to non-standard appended data after the end of the main JPEG image.) The XMP metadata identifies these extra JPEG images as the GainMap, Depth, and Confidence maps. Each of these images have their own EXIF data.
  
  ExifTool only displays the EXIF data for the main image. However, these parasites have their own EXIF data. Using the Strings analyzer at FotoForensics, you can see their EXIF dates. (Scroll to the bottom of the strings listing, then page-up about 3 times.) The data looks like:
  

  0x0030ed57: 2025:08:25 19:45:31
  
  0x0030ed8d: 0232
  
  0x0030ee35: 0100
  
  0x0030ef09: 2025:08:25 19:45:31
  
  0x0030ef1d: 2025:08:25 19:45:31
  
  0x0030ef31: -06:00
  
  0x0030ef39: -06:00
  
  0x0030ef41: -06:00

  This data says that the parasites were created at 2025-08-25 19:45:31 -06:00 (that's 2025-08-26 01:45:31 GMT). That is one second after the file was notarized. Moreover, while the C2PA's manifest excludes the main image's EXIF data, it includes these parasites and their EXIF data! This indicates that the parasites were created after the file was notarized by Google.

With photos, it's possible for the times to vary by a second. This is because the timestamps usually don't track fractions of a second. For example, if the picture was taken at 28.99 seconds and the file took 0.01 seconds to write, then the created and modified times might be truncated to 28 and 29 seconds. However, there is no explanation for the parasite's timestamp to be 3 seconds after the file was created, or any time after being notarized by the trusted timestamp provider.

Remember: this is not one of my forgeries. This is native to the camera, and I have no explanation for how Google managed to either post-date the parasites before notarizing, or generated the manifest after having the file notarized. This inconsistent metadata undermines the whole point of C2PA. When genuine Pixel 10 files look forged, investigators will conclude "tampering", even if the file is not manually altered.

With the Pixel 10's C2PA implementation, either the timestamps are untrustworthy, or the C2PA signatures are untrustworthy. But in either case, the recipient of the file cannot trust the data.

However, the problems don't stop there. Both of these sample pictures also include an MPF metadata field. The MPF data typically includes pointers to parasitic images at different resolutions. In the lamp picture, the MPF properly points to the Gain Map (a JPEG attached as a parasite). However, in these truck and graffiti examples, the MPF doesn't point to a JPEG. Typically, applications fail to update the MPF pointers after an alteration, which permits tamper detection. With these examples, we have clear indications of tampering: inconsistent metadata, inconsistent timestamps, evidence of post-dating or an untrusted signature, and a broken MPF. Yet, these are due to the camera app and Google's flawed implementation; they are not caused by a malicious user. Unfortunately, a forensic investigator cannot distinguish an altered Pixel 10 image from an unaltered photo.

Google Pixel 10: Now with Fraud Enabled by Default!

There's a very common insurance fraud scheme where someone will purchase their new insurance policy right after their valuable item is damaged or stolen. They will alter the date on their pre- and post-damage photos so that it appears to be damaged after the policy becomes active.

• Without C2PA, the insurance investigator will need to carefully evaluate the metadata in order to detect signs of alterations.


• With C2PA in a Google Pixel 10, the investigator still needs to evaluate the metadata, but now also needs to prove that the C2PA cryptographic signature from Google is meaningless.

Typical users might think that the cryptographic signature provides some assurance that the information is legitimate. However, the Pixel 10's implementation with C2PA is grossly flawed. (Both due to the Pixel 10 and due to C2PA.) There are no trustworthy assurances here.

Privacy Concerns

Beyond their inadequate implementation of the flawed C2PA technology, the Google Pixel 10 introduces serious privacy issues. Specifically, the camera queries Google each time a picture needs to be digitally signed by a trusted signing authority. Moreover, every picture taken on the Google Pixel 10 gets signed.

What can Google know about you?

• The C2PA signing process generates a digest of the image and sends that digest to the remote trusted timestamp service for signing. Because your device contacted Google to sign the image, Google knows which signature they provided to which IP address and when. The IP address can be used for a rough location estimation. Google may not have a copy of the picture, but they do have a copy of the signature.


• Since the Pixel 10 queries Google each time a photo is captured, Google knows how often you take pictures and how many pictures you take.


• While the C2PA metadata can be easily removed, the Pixel 10 reportedly also uses an invisible digital watermark called "SynthID". Of course, the details are kept proprietary because, as Google describes it, "Each watermarking configuration you use should be stored securely and privately, otherwise your watermark may be trivially replicable by others." This means, the only way to validate the watermark is to contact Google and send them a copy of the media for evaluation.

All of this enables user and content tracking. As I understand it, there is no option to disable any of it. If Google's web crawler, email, messaging system, etc. ever sees that signature again, then they know who originated the image, when and where it was created, who received a copy of the media and, depending on how Google acquired the data, when it was received.

With any other company, you might question the data collection: "While they could collect this, we don't know if they are collecting it." However, Google is widely known to collect as much user information as possible. While I have no proof, I have very little doubt that Google is collecting all of this information (and probably much more).

The inclusion of C2PA into the Pixel 10 appears to be more about user data collection and tracking than authenticity or provenance.

Security and Conformance

C2PA recently introduced their new conformance program. This includes two assurance levels. Level 1 has minimal security requirements, while Level 2 is supposed to be much more difficult to achieve and provides greater confidence in the information within the file.

There is currently only one device on the conforming products list that has achieved assurance level 2: The Google Pixel Camera. That's right, the same one that I just used to create an undetectable forgery and that normally generates inconsistent metadata.

The Provenance and Authenticity Standards Assessment Working Group (PASAWG) is performing a formal evaluation on C2PA. Some folks in the group posed an interesting theory: perhaps the Google Pixel Camera is compliant with assurance level 2. Since Google explicitly excludes everything about the hardware, they are technically conforming by omitting that information. Think of this like intentionally not attaching your bicycle lock to the entire bike. Sure, the bike can get stolen, but the lock didn't fail!

What if they fixed it?

You're probably wondering how something like this could happen at Google. I mean, regardless of whether you like the company, Google is usually known for cutting edge technology, high quality, and above-average security.

• Maybe this is just an implementation bug. Maybe nobody at Google did any kind of quality assurance testing on this functionality and it slipped past quality control.


• Maybe they were so focused on getting that "we use C2PA" and "Assurance Level 2" checkbox for marketing that they didn't mind that it didn't protect any of the metadata.


• Maybe nobody in Google's security group evaluated C2PA. This would certainly explain how they could put their corporate reputation on this flawed solution.


• Maybe nobody in Google's legal department was consulted about Google's liability regarding authenticating a forgery that could be used for financial fraud, harassment, or propaganda.

You might be thinking that Google could fix this if they didn't exclude the EXIF and XMP metadata from the cryptographic protection. (That would certainly be a step in the right direction.) Or maybe they could put some device metadata in the manifest for protection? However, you'd still be wrong. The C2PA implementation is still vulnerable to file system and hardware exploits.

These are not the only problems I've found with Google Pixel 10's C2PA implementation. For example:

• In the last few days, FotoForensics has received a handful of these pictures, including multiple pictures from the same physical device. As far as I can tell, Google uses the exact same four root certificates on every camera:
  
  
  • Google LLC, s/n 4B06ED7C78A80AFEB7193539E42F8418336D2F27
  • Google LLC, s/n 4FCA31F82632E6E6B03D6B83AB98B9D61B453722
  • Google LLC, s/n 5EF6120CF4D31EBAEAF13FB9288800D8446676BA
  • Google LLC, s/n 744428E3A7477CEDFDE9BD4D164607A9B95F5730
  
  I don't know why Google uses multiple root certs. It doesn't seem to be tied to the selected camera or photo options.
  
  While there are a limited number of root certs, every picture seems to use a different signing certificate, even if it comes from the same camera. It appears that Google may be generating a new signing certificate per picture. What this means: if a device is compromised and used for fraud, they cannot revoke the certificate for that device. Either they have to revoke a root cert that is on every device (revoking everyone's pictures), or they have to issue revocations on a per-photo basis (that doesn't scale).


• My associates and I have already identified easy ways to alter the timestamps, GPS information, and more. This includes ways that require no technical knowledge. The C2PA proponents will probably claim something like "The C2PA manifest don't protect that information!" Yeah, but tell that to the typical user who doesn't understand the technical details. They see a valid signature and assume the picture is valid.


• There's a physical dismantling (teardown) video on YouTube. At 12:27 - 12:47, you can see the cable for the front-facing camera. At 14:18 - 14:35 and 15:30 - 16:20, you can see how to replace the back-facing cameras. Both options provide a straightforward way for hardware hackers to feed in a false image signal for signing. With this device, the C2PA cryptographic signature excludes the metadata but covers the visual content. Unfortunately, you cannot inherently trust the signed image.


• Even if you assume that the hardware hasn't been modified, every picture has been tagged by Google as a composite image. That will impact insurance claims, legal evidence, and photo journalism. In fields where a composite image is not permitted, the Google Pixel 10 should not be used.

With Google's current implementation, their C2PA cryptographic signature is as reliable as signing a blank piece of paper. It doesn't protect the important information. But even if they fix their exclusion list, they are still vulnerable to C2PA's fundamental limitations. C2PA gives the appearance of authentication and provenance without providing any validation, and Google's flawed implementation just makes it worse. It's a snake oil solution that provides no meaningful information and no reliable assurances.

A lot of people are excited about the new Google Pixel 10. If you want a device that takes a pretty picture, then the Pixel 10 works. However, if you want to prove that you took the picture, value privacy, or plan to use the photos for proof or evidence, then absolutely avoid the Pixel 10. The cryptographic "proof" provided by the Pixel 10 is worse than having a device without a cryptographic signature. Every picture requires contacting Google, the unaltered metadata is inconsistent, the visual content is labeled as an AI-generated composite, the signed data may be post-dated, and there is no difference between an altered picture and an unaltered photo. I have honestly never encountered a device as untrustworthy as the Pixel 10.

A few months ago I wrote about my experimentation this year with solar power. I thought I would give a couple of updates.

The basic architecture hasn't changed, but some of the components have:



Given that I've never done this before, I expected to have some problems. However, I didn't expect every problem to be related to the power inverter. The inverter converts the 12V DC battery's power to 120V AC for the servers to use. Due to technical issues (none of which were my fault), I'm currently on my fourth power inverter.

Inverter Problem #1: "I'm Bond, N-G Bond"

The first inverter that I purchased was a Renogy 2000W Pure Sine Wave Inverter.



This inverter worked fine when I was only using the battery. However, if I plugged it into the automated transfer switch (ATS), it immediately tripped the wall outlet's circuit breaker. The problem was an undocumented grounding loop. Specifically, the three-prong outlets used in the United States are "hot", "neutral", and "ground". For safety, the neutral and ground should be tied together at one location; it's called a neutral-ground bond, or N-G bond. (For building wiring, the N-G bond is in your home or office breaker box.) Every outlet should only have one N-G bond. If you have two N-G bonds, then you have a grounding loop and an electrocution hazard. (A circuit breaker should detect this and trip immediately.)

The opposite of a N-G bond is a "floating neutral". Only use a floating neutral if some other part of the circuit has the N-G bond. In my case, the automated transfer switch (AFS) connects to the inverter and the utility/wall outlet. The wall outlet connects to the breaker box where the N-G bond is located.

What wasn't mentioned anywhere on the Amazon product page or Renogy web site is that this inverter has a built-in N-G bond. It will work great if you only use it with a battery, but it cannot be used with an ATS or utility/shore power.

There are some YouTube videos that show people opening the inverter, disabling the N-G bond, and disabling the "unsafe alarm". I'm not linking to any of those videos because overriding a safety mechanism for high voltage is incredibly stoopid.

Instead, I spoke to Renogy's customer support. They recommended a different inverter that has an N-G bond switch: you can choose to safely enable or disable the N-G bond. I contacted Amazon since it was just past the 30-day return period. Amazon allowed the return with the condition that I also ordered the correct one. No problem.

The big lesson here: Before buying an inverter, ask if it has a N-G bond, a floating neutral, or a way to toggle between them. Most inverters don't make this detail easy to find. (If you can't find it, then don't buy the inverter.) Make sure the configuration is correct for your environment.

• If you ever plan to connect the inverter to an ATS that switches between the inverter and wall/utility/shore power, then you need an inverter that supports a floating neutral.


• If you only plan to connect the inverter to a DC power source, like a battery or generator, then you need an inverter that has a built-in N-G bond.

Inverter Problem #2: It's Wrong Because It Hertz

The second inverter had a built-in switch to enable and disable the N-G bond. The good news it that, with the N-G bond disabled, it worked correctly through the ATS. To toggle the ATS, I put a Shelly Plug smart outlet between the utility/wall outlet and the ATS.



I built my own controller and it tracks the battery charge level. When the battery is charged enough, the controller tells the inverter to turn on and then remotely tells the Shelly Plug to turn off the wall outlet. That causes the ATS to switch over to the inverter.

Keep in mind, the inverter has it's own built-in transfer switch. However, the documentation doesn't mention that it is "utility/shore priority". That is, when the wall outlet has power, the inverter will use the utility power instead of the battery. It has no option to be plugged into a working outlet and to use the battery power instead of the outlet's power. So, I didn't use their built-in transfer switch.

This configuration worked great for about two weeks. That's when I heard a lot of beeping coming from the computer rack. The inverter was on and the wall outlet was off (good), but the Tripp Lite UPS feeding the equipment was screaming about a generic "bad power" problem. I manually toggled the inverter off and on. It came up again and the UPS was happy. (Very odd.)

I started to see this "bad power" issue about 25% of the time when the inverter turned on. I ended up installing the Renogy app to monitor the inverter over the built-in Bluetooth. That's when I saw the problem. The inverter has a frequency switch: 50Hz or 60Hz. The switch was in the 60Hz setting, but sometimes the inverter was starting up at 50Hz. This is bad, like, "fire hazard" bad, and I'm glad that the UPS detected and prevented the problem. Some of my screenshots from the app even showed it starting up low, like at 53-58 Hz, and then falling back to 50Hz a few seconds later.


(In this screenshot, the inverter started up at 53.9Hz. After about 15 seconds, it dropped down to 50Hz.)

I eventually added Bluetooth support to my homemade controller so that I could monitor and log the inverter's output voltage and frequency. The controller would start up the inverter and wait for the built-in Bluetooth to come online. Then it would read the status and make sure it was at 60Hz (+/- 0.5Hz) and 120V (+/- 6V) before turning off the utility and transferring the load to the inverter. If it came up at the wrong Hz, the controller would shut down the inverter for a minute before trying again.

It took some back-and-forth discussions with the Renogy technical support before they decided that it was a defect. They offered me a warranty-exchange. It took about two weeks for the inverter to be exchanged (one week there, one week back). The entire discussion and replacement took a month.

The replacement inverter was the same make and model. It worked great for the first two weeks, then developed the exact same problem! But rather than happening 25% of the time, it was happening about 10% of the time. To me, this looks like either a design flaw or a faulty component that impacts the entire product line. The folks at Renogy provided me with a warranty return and full refund.

If you read the Amazon reviews for the 2000W and 3000W models, they have a lot of 1-star reviews with comments about various defects. Other forums mention that items plugged into the inverter melted and motors burned out. Melting and burned out motors are problems that can happen if the inverter is running at 50Hz instead of 60Hz.

The Fourth Inverter

For the fourth inverter, I went with a completely different brand: a Landerpow 1500W inverter. Besides having what I needed, it also had a few unexpectedly nice benefits compared to the Renogy:

• I had wanted a 2000W inverter, but a 1500W inverter is good enough. Honestly, my servers are drawing about 1.5 - 2.5 amps, so this is still plenty of overkill for my needs. The inverter says it can also handle surges of up to 3000W, so it can easily handle a server booting (which draws much more power than post-boot usage).


• The documentation clearly specifies that the Landerpow does not have an N-G bond. That's perfect for my own needs.


• As for dimensions, it's easily half the size of the Renogy 2000W inverter. The Landerpow also weighs much less. (When the box first arrived, I thought it might be empty because it was so lightweight.)


• The Renogy has a built-in Bluetooth interface. In contrast, the Landerpow doesn't have built-in Bluetooth. That's not an issue for me. In fact, I consider Renogy's built-in Bluetooth to be a security risk since it didn't require a login and would connect to anyone running the app within 50 feet of the inverter.


• The Landerpow has a quiet beep when it turns on and off, nothing like Renogy's incredibly loud beep. (Renogy's inverter beep could be heard outside the machine room and across the building.) I view Landerpow's quiet beep as a positive feature.


• With a fully charged battery and with no solar charging, my math said that I should get about 5 hours of use out of the inverter:
  
  
  • The 12V, 100Ah LiFePO4 battery should provide 10Ah at 120V. (That's 10 hours of power if you're using 1 amp.)
  
  
  • There's a DC-to-AC conversion loss around 90%, so that's 9Ah under ideal circumstances.
  
  
  • You shouldn't use the battery below 20% or 12V. That leaves 7.2Ah usable.
  
  
  • I'm consuming power at a rate of about 1.3Ah at 120V. That optimistically leaves 5.5 hours of usable power.
  
  With the same test setup, none of the Renogy inverters gave me more than 3 hours. The Landerpow gave me over 5 hours. The same battery appears to last over 60% longer with the Landerpow. I don't know what the Renogy inverter is doing, but it's consuming much more battery power than the Landerpow.


• Overnight, when there is no charging, the battery equalizes, so the voltage may appear to change overnight. Additionally, the MPPT and the controller both run off the battery all night. (The controller is an embedded system requires 5VDC and the MPPT requires 9VDC; combined, it's less than 400mA.) On top of this, we have the inverter connected to the battery. The Landerpow doesn't appear to cause any additional drain when powered off. ("Off" means off.) In contrast, the Renogy inverter (all of them) caused the battery to drain by an additional 1Ah-2Ah overnight. Even though nothing on the Renogy inverter appears to be functioning, "off" doesn't appear to be off.


• The Renogy inverter required a huge surge when first starting up. My battery monitor would see it go from 100% to 80% during startup, and then settle at around 90%-95%. Part of this is the inverter charging the internal electronics, but part is testing the fans at the maximum rating. In contrast, the Landerpow has no noticeable startup surge. (If it starts when the battery is at 100% capacity and 13.5V, then it will still be at 100% capacity and 13.5V after startup.) Additionally, then Landerpow is really quiet; it doesn't run the fans when it first turns on.

The Renogy inverter cost over $300. The Landerpow is about $100. Smaller, lighter, quieter, works properly, consumes less power, and less expensive? This is just icing on the cake.

Enabling Automation

My controller determines when the inverter should turn on/off. With the Renogy, there's an RJ-11 plug for a wired remote switch. The plug has 4 wires (using telephone coloring, that's black, red, green, and yellow). The middle two wires (red and green) are a switch. If they are connected, then the inverter turns on; disconnected turns it off.

The Landerpow also has a four-wire RJ-11 connector for the remote. I couldn't find the pinout, but I reverse-engineered the switch in minutes.

The remote contains a display that shows voltage, frequency, load, etc. That information has to come over a protocol like one-wire, I2C (two wire), UART (one or two wire), or a three wire serial connection like RS232 or RS485. However, when the inverter is turned off, there are no electronics running. That means it cannot be a communication protocol to turn it on. I connected my multimeter to the controller and quickly found that the physical on/off switch was connected to the green-yellow wires. I wired that up to my controller's on/off relay and it worked perfectly on the first try.

I still haven't worked out the communication protocol. (I'll save that for another day, unless someone else can provide the answer.) At minimum, the wires need to provide ground, +5VDC power for the display, and a data line. I wouldn't be surprised if they were using a one-wire protocol, or using the switch wires for part of a serial communication like UART or RS485. (I suspect the four wires are part of a UART communication protocol: black=ground, red=+5VDC, green=data return, and yellow=TX/RX, with green/yellow also acting as a simple on/off switch for the inverter.)

Pictures!

I've mounted everything to a board for easy maintenance. Here's the previous configuration board with the Renogy inverter:



And here's the current configuration board with the Landerpow inverter:



You can see that the new inverter is significantly smaller. I've also added in a manual shutoff switch to the solar panels. (The shutoff is completely mounted to the board; it's the weird camera angle that makes it look like it's hanging off the side.) Any work on the battery requires turning off the power. The MPPT will try to run off solar-only, but the manual warns about running from solar-only without a battery attached. The shutoff allows me to turn off the solar panels before working on the battery.

Next on the to-do list:

• Add my own voltmeter so the controller can monitor the battery's power directly. Reading the voltage from the MPPT seem to be a little inaccurate.


• Reverse-engineering the communication to the inverter over the remote interface. Ideally, I want my own M5StampS3 controller to read the inverter's status directly from the inverter.

As components go, the Renogy solar panels seem very good. The Renogy MPPT is good, but maybe not the best option. Avoid Renogy inverters and consider the Landerpow inverter instead. I'm also a huge fan of Shelly Plugs for smart outlets and the M5StampS3 for the DIY controller.

Efficiency

Due to all of the inverter problems, I haven't had a solid month of use from the solar panels yet. We've also had a lot of overcast and rainy days. However, I have had some good weeks. A typical overcast day saves about 400Wh per day. (That translates to about 12kWh/month in the worst case.) I've only had one clear-sky day with the new inverter, and I logged 1.2kWh of power in that single day. (A month of sunny days would be over 30kWh in the best case.) Even with partial usage and overcast skies, my last two utility bills were around 20kWh lower than expected, matching my logs -- so this solar powered system is doing its job!

I've also noticed something that I probably should have realized earlier. My solar panels are installed as awnings on the side of the building. At the start of the summer, the solar panels received direct sunlight just after sunrise. The direct light ended abruptly at noon as the sun passes over the building and no longer hit the awnings. They generate less than 2A of power for the rest of the day through ambient sunlight.

However, we're nearing the end of summer and the sun's path through the sky has shifted. These days, the panels don't receive direct light until about 9am and it continues until nearly 2pm. By the time winter rolls around, it should receive direct light from mid-morning until a few hours before sunset. The panels should be generating more power during the winter due to their location on the building and the sun's trajectory across the sky. With the current "overcast with afternoon rain", I'm currently getting about 4.5 hours a day out of the battery+solar configuration. (The panels generate a maximum of 200W, and are currently averaging around 180W during direct sunlight with partially-cloudy skies.)

I originally allocated $1,000 for this project. With the less expensive inverter, I'm now hovering around $800 in expenses. The panels are saving me a few dollars per month. At this rate, they will probably never pay off this investment. However, it has been a great way to learn about solar power and DIY control systems. Even with the inverter frustrations, it's been a fun summer project.

Usually my parent/pal/people tech support duty (PTSD) blogs are about friends asking me for help. But sometimes I come across people who don't know enough to ask, or don't know who to ask. (Note: I'm not soliciting people to ask me for help.)

I watch the local news every evening. Most people watch it over cable TV, but I don't have cable. (I haven't subscribed to a cable TV service in over 25 years.) They do have broadcast news, but when everything moved from analog to digital in 2009, I stopped receiving that. So, I watch the news over their streaming service. They offer a live stream on their web site and through their Roku app.

I'm a huge fan of our local news service, 9News in Denver. They cover local things around the state, as well as the major national and international topics. Moreover, they report on positive, entertaining, and inspirational topics, not just the negative. (They are not like other news channels. I've seen some news stations that are nothing more than a list of deaths and body counts around the city, state, country, and world.) 9News is really good at sifting through everything and giving accurate fact-based reporting while not causing viewers to become steadily depressed.

9News has even made national news, such as when Kyle Clark moderated a 2024 political debate. The Columbia Journalism Review remarked, "That’s how you run a debate!", noting: "[Kyle Clark] refused to allow the candidates to evade his direct questions with waffling, rambly answers, instead repeatedly cutting them off: “You didn’t make any attempt to answer the actual question,” he said at one point." (Kyle also became internet-famous in 2013 for ranting about snow-covered patio photos.)

Keep in mind, it's not just Kyle Clark. Kim Christiansen, Jennifer Meckles, Jeremy Jojola, and other staffers have each earned several awards for news reporting. I don't mean to slight the others through omission; it's a long list of reporters and investigators. There are no slackers on their staff, and they are all held to a very high standard. Just by being on 9News, I trust the quality of their reporting. (And for my regular blog followers, you know that I don't typically have blind trust.)

Technical Difficulties

A few months ago, their newsroom was doing some upgrades that were causing technical problems. One day they had no video. Another day there was no audio. I mean, seriously, the news anchor used a paper flipboard to write out the news!



(Instead of calling it "9News", we jokingly called it "Mime News". Kyle Clark was the guy trapped in the TV box and couldn't make a sound.)

The Next! Problem

Fortunately, the audio problem only lasted for one broadcast. Unfortunately, it was followed by another problem: the live stream broke. For a few days, it wouldn't play at all. After a few days, it started up in 4x fast forward mode for a few seconds (without sound) before freezing completely. Meanwhile, I was writing in almost daily complaining that their Roku and live streaming services were not working. (I want to watch their news!)

After more than a week of this, the problem changed. I could see the video, but whenever it switched to or from a commercial break, the audio would drop. It wouldn't recover without restarting the stream. (You could either reload the web page or close and restart the Roku app. In either case, you'd miss the first 20-30 seconds of each news segment.)

My inquiries eventually got me in touch with their IT staff. Yes, they knew there was a problem. Unfortunately, it was inconsistent and not impacting everyone. Moreover, they were having trouble tracking down the cause. (As a techie, I can totally understand how hard it is to track down an inconsistent problem, especially when you cannot reproduce it locally.)

Well, this sounded like a job for "PTSD Man!" and his People Tech Support Duties!

Debugging a Stream

The first thing I did was check with a few friends who watch the same news using the same Roku app. One friend had the same "it's broken" problem. The other friend had no problem playing the newscast. (At least I could duplicate the "doesn't impact everyone" issue reported by the IT staff.)

Debugging the live stream wasn't easy. While some video streams are unencrypted, the news was being streamed over HTTPS. What this meant: I couldn't just use wireshark to sniff the stream and I couldn't directly detect the problem's cause.

I tried a different approach. If I couldn't see the streaming data directly, perhaps I could download fragments and identify any issues.

Chrome and Firefox have a developer panel that shows the web-based network requests. Unfortunately, it doesn't show the raw media streams. However, live streams typically have a series of web requests that contain URLs to the raw stream segments. I could see those requests in the developer panel and their the list of URLs. A typical sequence reply might look like:

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-TARGETDURATION:6
#EXT-X-MEDIA-SEQUENCE:7800004
#EXT-X-DISCONTINUITY-SEQUENCE:0
#EXT-X-PROGRAM-DATE-TIME:2025-08-20T18:52:06.336Z
#EXTINF:6.006,
https://playback.tegnaone.com/kusa/live/index_3_7800004.ts?m=1716401672
#EXT-X-PROGRAM-DATE-TIME:2025-08-20T18:52:12.342Z
#EXTINF:6.006,
https://playback.tegnaone.com/kusa/live/index_3_7800005.ts?m=1716401672
#EXT-X-PROGRAM-DATE-TIME:2025-08-20T18:52:18.348Z
#EXTINF:6.006,
https://playback.tegnaone.com/kusa/live/index_3_7800006.ts?m=1716401672
#EXT-X-PROGRAM-DATE-TIME:2025-08-20T18:52:24.354Z
...

These URLs to the raw MPEG stream data can be easily requested with wget. I grabbed a few samples during the newscast and during commercials. Poof -- I found the problem.

There are many different ways to encode a video stream. There's not just one compression setting; there are lots of choices. A video may use a constant bitrate (CBR) or variable bitrate (VBR). The frame rate can also be constant or variable (CFR or VFR). At least, that's the theory.

In practice, there are some things that should never change and some things always change. For example:

• With video (the visual portion), the aspect ratio should never change. (This doesn't mean it doesn't, but it shouldn't.)


• VBR is very common with most audio codecs. Some, like the Advanced Audio Coding (AAC) method (which is extremely common), almost always uses VBR. MPEG supports VBR, but CBR is common.


• VFR for video is often used if the video has long segments that don't change. ("Long" could be a fraction of a second or minutes.) This way, they only encode the one frame and leave it on the screen, rather than re-encoding and transmitting the same data over and over.


• VFR for audio is very uncommon because it can cause synchronization errors. Moreover, audio VFR can result in audio misalignment if you try to fast-forward or rewind the audio stream. (If you've ever fast-forwarded or rewound a video and the audio was out of sync for a few seconds, that could be VFR or just bad alignment.)


• While the bitrate and frame rate may change in the stream, the sample rate is usually constant. For audio, MPEG-1 uses a fixed sample rate and does not support changing the rate within a single stream. WAV and PCM only define the rate once, so it cannot change. AAC does support a variable rate, but it's uncommon; a fixed sample rate is typical. Moreover, some AAC profiles (like the kind typically used for streaming broadcasts) does not support a variable sample rate.

The FFMpeg media library doesn't always work well with a variable sample rate. Depending on the codec, the library might detect a change in the rate and decide the stream is corrupted. Worse: if the stream is associated with the video, then the corruption may cause unexpected results, like playing at the wrong speed or hanging. If the sample rate changes in the audio stream, then the player may just assume the audio is corrupted and stop playing sound.

That's what I was seeing with the live newsfeed. They were changing rates between the shows and the commercials. The change was detected as a corruption and the stream would drop sound.

Keep in mind, not all media players do this. It depends on the player and library version. (And as the user, you probably don't know what you're using.) Some libraries see the change, flush the buffer, and can safely recover from the corruption. However, other libraries see the corruption and give up. This makes the problem inconsistent between different people and different media players.

News to You

I reported my findings to the news channel's IT staff. They went running off and had the problem fixed in under 30 minutes. It's worked flawlessly since. (However, if you use Wireshark, you can see a ton of out-of-order TCP packets and retries, so I think they still have a networking problem. But that's probably due to the CDN and not the IT staff.) Today, I can watch the news again via Roku or in my web browser. (Huge thanks to the IT staff for listening to a rando spouting technical details over email, and for being incredibly responsive as soon as the problem was explained.)

On various online "can anyone help" forums, there are a lot of people reporting similar streaming problems with other online streaming services. I suspect they are all the same problem: the stream providers are changing the sample rate incorrectly, changing the aspect ratio (never change the aspect ratio in a video stream!), or otherwise failing to normalize the media between different segments. This is causing the media library to detect a corruption and the stream fails.

Now for the Bad News

I'm thrilled to be able to watch the news again via the streaming services. Unfortunately, earlier this week it was announced that the local Denver NBC affiliate's parent company, TEGNA, is being sold to Nexstar. Nexstar owns the local FOX station.

Personally, I equate FOX with fiction, conspiracies, and propaganda. This goes along with FOX repeatedly being involved in defamation and false reporting lawsuits, such as paying $758M to settle with Dominion Voting over FOX's false reporting, being sued for $2.7 Billion by Smartmatic, and most recently (June 2025) being sued by California's Governor Newsom for alleged false reporting. (Friends don't let friends watch FOX.)

In contrast to FOX, I think our local NBC 9News provides fair and balanced reporting. To me, they epitomize journalistic integrity. I don't know if they will continue that way after the merger and restructuring, or if we will have one less reliable news source in the area and the world.

Even though I don't know them personally, their newscasts come into my home every evening. They're so regular, that they feel like part of my extended family. (And like my extended family, I'm glad they don't regularly visit in person.) I typically reserve tech support for family and friends, which is why the folks at 9News became my newest cause of PTSD. If the staff at 9News end up jumping ship to another station, I'm certain to follow them.

I've spent the last few months evaluating different AI sound generation systems. My original question was whether I could detect AI speech. (Yup.) However, this exploration took me from AI voices to AI videos and eventually to AI music. The music aspect strikes me as really interesting because of the different required components that are combined. These include:

• Score: The written music that conveys the tune, melody, harmony, instruments, etc. This is the documentation so that other musicians can try to replay the same music. But even with unwritten music, something has to come up with the melody, harmony, composition, etc.


• Instrumentation: The score doesn't contain everything. For example, if the music says it is written for a piano, there are still lots of different types of pianos and they all have different sounds. The instrumentation is the selection of instruments and when they play.


• Lyrics: The written words that are sung.


• Vocals: The actual singing.

This is far from everything. I'm not a musician; real musicians have their own terminologies and additional breakdowns of these components.

Each of these components have their own gray levels between AI and human. For example, the vocals can be:

• AI: Completely generated using AI.


• Real: Completely human provided.


• Augmented: A human voice that is adjusted, such as with autotune or other synthetic modulators. (Like Cher singing "Believe", or almost anything from Daft Punk.)


• Synthetic: From a synthesizer -- artificially created, but not AI. This could be a drum machine, a full synthesizer, like Moog or Roland, or even a Yamaha hybrid piano with built-in background rhythm player like Rowlf the Dog uses. (The Eurythmics is a good example of a music group whose earlier works were heavily dependent on synthesizers.)


• Human edited: Regardless of the source, a human may edit the vocals during post-production. For example, Imogen Heap's "Hide and Seek" loops the human's voice at different pitches to create a layered vocal harmony. And Billy Joel's "The Longest Time" features multiple singing voices that are all Billy Joel. He recorded himself singing each part, then combined them for the full song.


• AI edited: Regardless of the source, an AI system may edit the vocals during post-production. Tools that can do this include Audimee and SoundID VoiceAI. (Not an endorsement of either product.) Both can generate harmonies from single voice recordings.

The same goes for the score, arrangement, and even the lyrics. There isn't a clear line between "human" and artificial creations. Unless it's a completely live performance (acoustic and unplugged), most music is a combination.

Detecting AI (Just Beat It!)

Detecting whether something is AI-generated, synthetic, or human -- and the degree of each combination -- can be really difficult. Currently, different AI systems have different 'tells' that can be detected. Each AI system seems to have different quirks. However, each generation changes the detectable artifacts and there may be a new generation every few months. In effect, detection is a moving target.

Because this is a rapidly changing field, I'm not too concerned about giving away anything by disclosing detection methods. Any artifacts that I can detect today are likely to change during the next iteration.

Having said that, it seems relatively easy to differentiate between AI, synthetic, and human these days. Just consider the music. An easy heuristic relies on the "beat":

• No matter how good a human musician is, there are always micro-variations in the beat. The overall song may be at 140 bpm (beats per minute), but the tempo at any given moment may be +/- 5 bpm (or more).
  
  For example, I graphed the beats over time for the song "Istanbul (Not Constantinople)" from They Might Be Giants. This comes from their live album: Severe Tire Damage:
  
  
  
  
  The red lines at the bottom identifies each detected beat, while the blue line shows a ten-second running average to determine the beats per minute. This version of the song starts with a trumpet solo and then they bring in the drums. The trumpet appears as a very unsteady rhythm, while the drums are steadier, but show a continual waver due to the human element.
  
  As another example, here's Tears for Fears singing "Everybody Wants to Rule the World":
  
  
  
  
  Even though they use a lot of synthetic music elements, Tears for Fears combines the synthetic components with humans playing instruments. The human contribution causes the beats per minute to vary.


• Synthetic music, or music on a loop, is incredibly consistent. The only times it changes is when a human changes up the loop or melody. For example, this is Rayelle singing "Good Thing Going":
  
  
  
  
  Even though the vocals are from a human, the music appears synthetic due to the consistency. There's a slight change-up around 2 minutes into the song, where the music switches to a different loop.
  
  Another incredible example is "Too High" by Neon Vines:
  
  
  
  
  The music is completely synthetic. You can see her creating the song in the YouTube video. Initially, she manually taps the beat, but then it loops for a consistent 109 bpm. At different points in the song, she manually adds to the rhythm (like at the 1 minute mark), adding in a human variation to the tempo.
  
  Many professional musicians often record to an electronic "click track" or digital metronome that can help lock-in the beat, which also makes the music's rhythm extremely consistent. The song "Bad Things" by Jace Everett is a perfect example:
  
  
  
  
  
  If you listen to the music, you can clearly hear the electronic percussion keeping time at 132 bpm. There may have also been some post-production alignment for the actual drums. (No offense to the drummer.)


• AI music systems, like Suno or Udio, have a slow variation to the beat. It varies too much to be synthetic, but not enough to be human. For example, "MAYBE?!" by Around Once uses Suno's AI-generated music:
  
  
  
  
  The beat has a slow variation that doesn't change with the different parts of the song (verse, bridge, chorus, etc.). This is typical for AI generated music (not just Suno).

The distinction between "synthetic" and "AI" music is not always clear. Modern synthesizers often use embedded AI models for sound design or generative sequencing, which blurs the boundaries. Moreover, post-processing for beat alignment against a click-track can make a real performance appear as synthetic or AI.

By evaluating the beat over time, the music can be initially classified into human, synthetic, or AI-generated. (And now that I've pointed that out, I expect the next version of these AI systems to make it more difficult to detect.)

Lyrics and AI (Word Up!)

Lyrics are equally interesting because of how the different approaches combine words. For example:

• ChatGPT can compose lyrics. However, the model I tested usually drops "g" from "ing" words (e.g., "drivin'" instead of "driving") and uses lots of m-dashes for pauses. (Real writers use m-dashes sparingly.) When mentioning women, it regularly uses "baby" or "honey". (Because those are the only two terms of endearment in the English language, right?) It also seems incapable of repeating a verse without changing words. As far as intricacy goes, ChatGPT is great at including subtle elements such as emotion or innuendo. (Note: I tested against ChatGPT 4. Version 5 came out a few days ago.)


• Suno has two lyric modes: the user can provide their own lyrics, or it can generate lyrics from a prompt. When using the prompt-based system, the songs are usually short (six structures, like verses, bridge, and chorus). As for linguistics, it seems to prefer night/evening language over day/morning and uses the specific word "chaos" (or "chaotic") far too often.
  
  Unlike ChatGPT, Suno is great at repeating verses, but it sometimes ends the song with partial repetition of previous verses, weird sounds, or completely different musical compositions. The AI-generated score, instrumentation, and vocal components don't seem to known when to end the song, and may not follow the written lyrics to the letter. The free Suno version 3.5 does this often; the paid version 4.5 does it much less often, but still does it.


• Microsoft Copilot? Just don't. It writes really primitive short songs with bad wording and inconsistent meter.


• Gemini is a slight improvement over Copilot. The lyrics are often longer and have better rhyming structure, but lack any subtly or nuance.

In every case, the AI-generated lyrics often include poor word choices. This could be because the AI doesn't understand the full meaning of the word in context, or because it chose the word's sound over the meaning for a stronger rhyme. (If you're going to use AI to write a song, then I strongly suggest having a human act as a copy editor and fix up the lyrics.)

If you need a fast song, then try using ChatGPT to write the initial draft. However, be sure to use a human to edit the wordings and maybe replace the chorus. Then use Suno to put it to music and generate the AI vocals.

During my testing, I also evaluated songs from many many popular artists. From what I can tell, some recent pop songs appear to do just that: it sounds like ChatGPT followed by human edits. A few songs also seem to use some AI music to create the first draft of the song, then used humans and/or synthesizers to recreate the AI music so it becomes "human made". Basically, there are some composition choices that AI systems like to make that differ from human-scored music. (I'm not going to name names because the music industry is litigious.)

Ethics and AI (Should I Stay or Should I Go?)

I'm not sure how I feel about AI-generated music. On one hand, a lot of AI-generated music is really just bad. (Not 'bad' like 'I don't like that genre' or song, but 'bad' as in 'Turn it off', 'My ears are bleeding', and 'The lyrics make no sense'.) Even with AI assistance, it's hard to make good music.

On the other hand, not everyone is a musician or can afford a recording studio. Even if you have a good song in your head, you may not have the means to turn it into a recording. AI-generated music offers the ability for less-talented people (like me) to make songs.

The use of AI to make creative arts is very controversial and strong concerns are found in the AI-generated artwork field. However, copying an artist's style (e.g., "in the artistic style of Larry Elmore") is not the same as saying "in the musical genre of 80s Hair Metal". One impersonates a known artist and impedes on their artistic rights, while the other is a generalized artistic style. (In copyright and trademark law, there have been "sound-alike" disputes with style emulation. So it's not as simple as saying that it's fine to stick to a genre. A possible defense is to show that everyone copies everyone and then find a similar rift from a much older recording that is out of copyright.)

AI-generated music is also different from AI-books. I'm not seeing a flood of AI-created crap albums flooding online sellers. In contrast, some book sellers are drowning in AI-written junk novels. The difference is that you often can't tell the quality of a book's contents without reading it, while online music sellers often includes a preview option, making it relatively easy to spot the bad music before you buy it.

Unlike artwork or books, music often has cover performances, where one band plays a song that was originally written by someone else. For example:

• They Might Be Giants' version of "Istanbul (Not Constantinople)" (1990) was a cover of a song originally made famous in 1953 by Jimmy Kennedy, but dates back to 1928's Paul Whiteman and His Orchestra.


• Elvis Presley's "Hound Dog" (1956) is based on a 1952 version by Big Mama Thornton. (I prefer Big Mama's version.)


• And let's not forget Hayseed Dixie's A Hillbilly Tribute to AC/DC. They recreated some of AC/DC's rock classics as country music -- same lyrics, similar melody, but different pace and instrumentation. (No offense to AC/DC, but I like the hillbilly version of "You Shook Me All Night Long" much more than the original.)

I don't see AI-music as "creating more competition for real musicians". Between cover bands and the wide range of current alterations permitted in music (autotune, beat machines, synthesizers, mashups, sampling, etc.), having completely AI-generated music and vocals seems like a logical next step. Rather, I view AI-music as way for amateurs, and the less musically talented, to create something that could pass for real music.

Bad Poetry and AI (Where Did My Rhythm Go?)

I was told by a friend who is a musician that songs often start with the melody and then they write lyrics to fit the music. However, many AI-music systems flip that around. They start with the lyrics and then fit the melody to the tempo of the lyrics. It's an interesting approach and sometimes works really well. This means: if you can write, then you can write a song. (Not necessarily a good song, but you can write a song.)

I'm definitely not a musician. I joke that I play a mean kazoo. (A kazoo doesn't have to be on key or play well with others.) As for lyrics, well, I tell people that I'm an award-winning professional poet. While that's all true, it's a little misleading:

• I've entered two limerick contests and won both of them. (Thus, "award winning.")


• One of my winning limericks was published in the Saturday Evening Post. (Thus, published.)


• And the published poem paid me $25! (Paid for my poem? That makes me a professional!)

With these AI systems, I quickly learned that a formal poetic structure really doesn't sound good when put to music. The uniformity of syllables per line makes for a boring song. For a good song, you really need to introduce rhythm variations and half-rhymes. (In my opinion, a good song is really just a bad poem put to music.)

During my testing, I listened to lots of real music (from professional musicians) as well as AI-generated music (from amateurs). I also created a ton of test examples, using a variety of lyric generation methods and genres. (Yes, the beat detector holds up regardless of the genre, singing voice, etc.) Let me emphasize: most of my test songs were crap. But among them were a few lucky gems. (Well, at least I enjoyed them.) These include songs with some of my longer poems put to music. (I wrote the poems while creating controlled test samples for human vs AI lyric detection.)

Having said that, I put some of my favorite test songs together into two virtual albums: Thinning the Herd and Memory Leaks. One of my friends suggested that I needed a band name ("Don't do this as Hacker Factor"), so "Brain Dead Frogs" was born.



I have two albums on the web site. They span a variety of genres and techniques. All of them use AI-generated music and AI voices (because I can't play music and I can't sing on key); only the lyrics vary: some are completely human written, some are based on AI creations but human edited, a few use AI-editing a human's writings, and one is completely written by AI. If you listen to them, can you tell which is which?

Hint: When editing, even if I changed all of the lyrics, I tried to keep the original AI artifacts. I found it was easiest to use AI to create the first draft of the song -- because that creates the pacing and tempo. Then I'd rewrite the lyrics while retaining the same syllables per line, which kept the pacing and tempo.

Outro (Thank You For The Music)

As a virtual musician and amateur songwriter, I'd like to thank everyone who helped make these songs possible. That includes Suno, ChatGPT, Gemini, Audacity, the online Merriam-Webster dictionary (for help with rhyming and synonyms), and Madcat for temporarily suspending her devotion to Taylor Swift long enough to be my first fan and website developer.

I definitely must thank my critical reviewers, including Bob ("I had low expectations, so this was better than I thought."), Richard ("Have you thought about therapy?"), The Boss ("At least it's not another C2PA blog"), and Dave for his insights about music and pizza. I'd also like to thank my muses, including Todd (nerdcore genre) and Zach Weinersmith for his cartoon about the fourth little pig (genre: traditional ska; a calypso and jazz precursor to reggae).

Finally, I'd like to issue a preemptive apology. For my attempt at K-pop ("Jelly Button"), I thought it needed some Korean in the bridge, because you can't have K-pop without some Korean lyrics. I don't speak Korean, and I suspect that my translator (ChatGPT) and singer (Suno) also don't. So I'll apologize in advance if the words are wrong. (But if the words are funny or appropriate, I'll unapologize and claim it was intentional.)

Last month I released my big bulleted list of C2PA problems. Any one of these issues should make potential adopters think twice. But 27 pages? They should be running away!

Since then, my list has been discussed at the biweekly Provenance and Authenticity Standards Assessment Working Group (PASAWG). The PASAWG is working on an independent evaluation of C2PA. Myself and the other attendees are only there as resources. As resources, we're answering questions and discussing issues, but not doing their research. (We've had some intense discussions between the different attendees.) The PASAWG researchers have not yet disclosed their findings to the group, and as far as I can tell, they do not agree with me on every topic. (Good! It means my opinion is not biasing them!)

Full disclosure: The PASAWG meetings fall under Chatham House rules. That means I can mention the topics discussed, but not attribute the information to any specific person without permission. (I give myself permission to talk about my own comments.)

Clarifications

For the last few meetings, we have been going over topics related to my bulleted list, the associated issues, and clarifying what the C2PA specification really provides. Having said that, I have found nothing that makes me feel any need to update my big bulleted list, except maybe to add more issues to it. There are no inaccuracies or items needing correction. The 27 pages of issues are serious problems.

However, I do want to make a few clarifications.

First, I often refer to C2PA and CAI as "All Adobe All The Time". One of my big criticisms is that both C2PA and CAI seem to be Adobe-driven efforts with very little difference between Adobe, C2PA, and CAI. I still have that impression. However:

• The C2PA organization appears to be a weak coalition of large tech companies. Adobe is the primary driving force, and to my knowledge, there are no C2PA working groups that do not have any Adobe employee as the chair or co-chair. The only exception is the top-level C2PA organization -- it's chaired by a Microsoft employee who is surrounded by Adobe employees. I refer to their structure as a "weak coalition" because Adobe appears to be the primary driving force.


• The Content Authenticity Initiative (CAI) is not "looks like Adobe". No, it is Adobe. Owned, managed, operated, and all code is developed by Adobe employees as part of the Adobe corporation. When you visit the CAI's web site or Content Credentials web service, that's 100% Adobe.

It's the significant overlap of Adobe employees who are part of both C2PA and CAI that causes a lot of confusion. Even some of the Adobe employees mix up the attribution, but they are often corrected by other Adobe employees.

The second clarification comes from the roles of C2PA and CAI. C2PA only provides the specification; there is no implementation or code. CAI provides an implementation. My interpretation is that this is a blame game: if something is wrong, then the C2PA can blame the implementation for not following the specs, while the implementers can blame the specification for not being concise or for any oversights. (If something is broken, then both sides can readily blame the other rather than getting the problem fixed.)

The specifications dictate how the implementation should function. Unless there is a bug in the code, it's a specification issue and not a programming problem. For example, my sidecar swap exploit, which permits undetectable alterations of the visual image in a signed file, is made possible by the specification and not the implementation. The same goes for the new C2PA certificate conformance (which is defined but not implemented yet); choosing to use a new and untested CA management system is a risk from the spec and not the implementation.

The third clarification comes from "defined but not implemented yet". Because C2PA does not release any code, everything about their specification is theoretical. Moreover, the specification is usually 1-2 revisions ahead of any implementations. This makes it easy for C2PA to claim that something works since there are no implementation examples to the contrary. By the time there are implementations that demonstrate the issues, C2PA has moved on to newer requirements and seems to disregard previous findings. However, some of the specification's assumptions are grossly incorrect, such as relying on technologies that do not exist today. (More on that in a moment.)

New Changes: v2.2

The current specification, v2.2, came out a few months ago. Although my bulleted list was written based on v2.1 and earlier, the review was focused on v2.2. When I asked who supports v2.2, the only answer was "OpenAI". Great -- they're a signer. There are no tools that can fully validate v2.2 yet. But there is some partial support.

I've recently noticed that the Adobe/CAI Content Credentials web site no longer displays the embedded user attribution. For example, my Shmoocon forgery used to predominantly display the forged name of a Microsoft employee. However, last month they stopped displaying that. In fact, any pictures (not just my forgeries) that include a user ownership attribution are no longer displayed. This is because the Content Credentials web site is beginning to include some of the C2PA v2.2 specification's features. The feature? User's names are no longer trusted and are now no longer displayed.

That's right: all of those people who previously used C2PA to sign their names will no longer have their names displayed because their names are untrusted. (While the C2PA and Adobe/CAI organizations haven't said this, I think this is in direct response to some of my sample forgeries that included names.)

If you dive into the specifications, there's been a big change: C2PA v2.0 introduced the concepts of "gathered assertions" and "created assertions". However, these concepts were not clearly defined. By v2.2, these became a core requirement. Unfortunately, trying to figure out the purpose and definitions from the specs is as clear as mud. Fortunately, the differences were clarified at the PASAWG meetings. The risks and what can be trusted basically breaks down to gathered assertions, created assertions, trusted certificates, and reused certificates.

Risk #1: Gathered assertions
Gathered assertions cover any metadata or attribution that comes from an unvetted source, such as a user entering their name, copyright information, or even camera settings from unvetted devices. Because the information is unverified, it is explicitly untrusted.

When you see any information under a gathered assertion, it should be viewed skeptically. In effect, it's as reliable as existing standard metadata fields, like EXIF, IPTC, and XMP. (But if it's just as reliable as existing standards, then why do we need yet-another new way to store the same information?)

Risk #2: Created assertions
Created assertions are supposed to come from known-trusted and vetted hardware. (See the "C2PA Generator Product Security Requirements", section 6.1.2.) However, there is currently no such thing as trusted hardware. (There's one spec for some auto parts that describes a trusted camera sensor for the auto industry, but the specs are not publicly accessible. I can find no independent experts who have evaluated these trusted component specs, no devices use the specs right now, and it's definitely not available to general consumers. Until it's officially released, it's vaporware.) Since the GPS, time, camera sensor, etc. can all be forged or injected, none of these created assertions can be trusted.

This disparity between the specification's theoretical "created assertions" and reality creates a big gap in any C2PA implementation. The specs define the use of created assertions based on trusted hardware, but the reality is that there are no trusted hardware technologies available right now. Just consider the GPS sensor. Regardless of the device, it's going to connect to the board over I2C, UART, or some other publicly-known communication protocol. That means it's a straightforward hardware modification to provide false GPS information over the wire. But it can be easier than that! Apps can provide false GPS information to the C2PA signing app, while external devices can provide false GPS signals to the GPS receiver. Forging GPS information isn't even theoretical; the web site GPSwise shows real-time information (mostly in Europe) where GPS spoofing is occurring right now.



And that's just the GPS sensor. The same goes for the time on the device and the camera's sensor. A determined attacker with direct hardware access can always open the device, replace components (or splice traces), and forge the "trusted sensor" information. This means that the "created assertions" that denote what was photographed, when, and where can never be explicitly trusted.



Remember: Even if you trust your hardware, that doesn't help someone who receives the signed media. A C2PA implementation cannot verify that the hardware hasn't been tampered with, and the recipient cannot validate that trusted hardware was used.

Requiring hardware modifications does increase the level of technical difficulty needed to create a forgery. While your typical user cannot do this, it's not a deterrent for organized crime groups (insurance and medical fraud are billion-dollar-per-year industries), political influencers, propaganda generators, nation-states, or even determined individuals. A signed cat video on Tick Tack or Facegram may come from a legitimate source. However, if there is a legal outcome, political influence, money, or reputation on the line, then the signature should not be explicitly trusted even if it says that it used "trusted hardware".

Risk #3: Trusted Certificates
The C2PA specification uses a chain of X.509 certificates. Each certificate in the chain has two components: the cryptography (I have no issues with the cryptography) and the attribution about who owns each certificate. This attribution is a point of contention among the PASAWG attendees:

• Some attendees believe that, as long as the root is trusted and we trust that every link in the chain follows the defined procedure of validating users before issuing certificates, then we can trust the name in the certificate. This optimistic view assumes that everyone associated with every node in the chain was trustworthy. Having well-defined policies, transparency, and auditing can help increase this trust and mitigate any risks. In effect, you can trust the name in the cert.


• Other attendees, including myself, believe that trust attenuates as each new node in the chain is issued. In this pessimistic view, you can trust a chain of length "1" because it's the authoritative root. (We're assuming that the root certs are trusted. If that assumption is wrong, then nothing in C2PA works.) You can trust a length of "2" because the trusted root issued the first link. But every link in the chain beyond that cannot be fully trusted.

This pessimistic view even impacts web certificates. HTTPS gets around this trust attenuation by linking the last node in the chain back to the domain for validation. However, C2PA's certificates do not link back to anywhere. This means that we must trust that nobody in the chain made a mistake and that any mistakes are addressed quickly. ("Quickly" is a relative term. When WoSign and StartCom were found to be issuing unauthorized HTTPS certificates, it took years for them to be delisted as trusted CA services.)

In either case, you -- as the end user -- have no means to automatically validate the name in the signing certificate. You have to trust the signing chain.

As an explicit example, consider the HTTPS certificate used by TruePic's web site. (TruePic is a C2PA steering committee member). When you access their web site, their HTTPS connection currently uses a chain of three X.509 certificates:

1. The root certificate is attributed to the Internet Security Research Group (ISRG Root X1). I trust this top level root certificate because it's in the CCADB list that is included in every web browser. (To be in the CCADB, they had to go through a digital colonoscopy and come out clean.)


2. The second certificate is from Let's Encrypt. Specifically, ISRG Root X1 issued a certificate to Let's Encrypt's "R11" group. It's named in the cert. Since I trust Root X1, I assume that Root X1 did a thorough audit of Let's Encrypt before issuing the cert, so I trust Let's Encrypt's cert.


3. Let's Encrypt then issued a cert to "www.truepic.com". However, their vetting process is really not very sophisticated: if you can show control over the host's DNS entry or web server, then you get a cert. In this case, the certificate's common name (CN) doesn't even name the company -- it just includes the hostname. (This is because Let's Encrypt never asked for the actual company name.) There is also no company address, organization, or even a contact person. The certificate has minimum vetting and no reliable attribution. If we just stop here, then I wouldn't trust it.
   
   However, there's an extra field in the certificate that specifies the DNS name where the cert should come from. Since this field matches the hostname where I received the cert (www.truepic.com), I know it belongs there. That's the essential cross-validation and is the only reason the cert should be trusted. We can't trust the validation process because, really, there wasn't much validation. And we can't trust the attribution because it was set by the second-level issuer and contains whatever information they wanted to include.

With web-based X.509 certificates, there is that link back to the domain that provides the final validation step. In contrast, C2PA uses a different kind of X.509 certificate that lacks this final validation step. If the C2PA signing certificate chain is longer than two certificates, then the pessimistic view calls the certificate's attribution and vetting process into question. The basic question becomes: How much should you trust that attribution?

Risk #4: Reused Certificates
Most services do not have user-specific signing certificates. For example, every picture signed today by Adobe Firefly uses the same Adobe certificate. The same goes for Microsoft Designer (a Microsoft certificate), OpenAI (a certificate issued by TruePic), and every other system that currently uses C2PA.

The attribution in the signature identifies the product that was used, but not the user who created the media. It's like having "Nike" on your shoes or "Levi's" on your jeans -- it names the brand but doesn't identify the individual. Unless you pay to have your own personalized signing certificate, the signature is not distinct to you. This means that it doesn't help artists protect their works. (Saying that the painter used Golden acrylic paint with a brush by Winsor & Newton doesn't identify the artist.)

As an aside, a personalized signing certificate can cost $50-$300 per year. Given all of C2PA's problems, you're better off using the US Copyright Office. They offer group registration for photographers: $55 for 750 photos per year, and the protection lasts for 70 years beyond the creator's lifetime. This seems like a more cost-effective and reliable option than C2PA.

Missing Goals

Each of these risks with C2PA pose serious concerns. And this is before we get into manifest/sidecar manipulations to alter the visual content without detection, inserting false provenance information, competing valid signatures, reissuing signatures without mentioning changes, applying legitimate signatures to false media, etc. Each of these exploits are independent of the implementation, and are due to the specifications.

The C2PA documentation makes many false statements regarding what C2PA provides, including:

• Section 3, Core Principles: "Content Credentials provides a way to establish provenance of content."


• Section 5.1: "Helping consumers check the provenance of the media they are consuming."


• Section 5.2: "Enhancing clarity around provenance and edits for journalistic work."


• Section 5.3: "Offering publishers opportunities to improve their brand value." (Except that the end consumer cannot prove that it came from the publishers.)


• Section 5.4: "Providing quality data for indexer / platform content decisions."

This is not the entire list of goals. (I'm literally going section by section through their document.) Unfortunately, you cannot have reliable provenance without validation. C2PA lacks attribution validation so it cannot meet any of these goals. C2PA does not mitigate the risk from someone signing content as you, replacing your own attribution with a competing claim, or associating your valid media with false information (which is a great way to call your own legitimate attribution into question).

What Does C2PA Provide?

An independent report came out of the Netherlands last month that reviews C2PA and whether it can help "combat disinformation by ensuring the authenticity of reporting through digital certificates." (Basically, it's to see if C2PA is appropriate for use by media outlets.) This report was commissioned by NPO Innovatie (NPO), Media Campus NL, and Beeld & Geluid. The report is written in Dutch (Google Translate works well on it) and includes a summary in English. Their key findings (which they included with italics and bold):

C2PA is a representation of authenticity and provenance, but offers no guarantee of the truth or objectivity of the content itself, nor of the factual accuracy of the provenance claims within the manifest.

(Full disclosure: They interviewed many people for this report, including me. However, my opinions are not the dominant view in this report.)

C2PA does not provide trusted attribution information and it provides no means for the end recipient to automatically validate the attribution in the signing certificate. Moreover, the specifications depend on trusted hardware, even though there is no such thing as trusted hardware. This brings up a critical question: If you cannot rely on the information signed using C2PA, then what does C2PA provide?

My colleague, Shawn Masters, likens C2PA's signature to an "endorsement". Like in those political ads, "My name is <name>, and I approve this message." You, as the person watching the commercial, have no means to automatically validate that the entity mentioned in the promotion actually approved the message. (An example of this false attribution happened in New Hampshire in 2024, where a deep fake robocall pretended to be Joe Biden.) Moreover, the endorsement is based on a belief that the information is accurate, backed by the reputation of the endorser.

The same endorsement concept applies to C2PA: As the recipient of signed media, you have no means to automatically validate that the name in the signing cert actually represents the cert. The only things you know: (1) C2PA didn't validate the content, (2) C2PA didn't validate any gathered assertions, and (3) the signer believes the unverifiable created assertions are truthful. When it comes to authenticating media and determining provenance, we need a solution that provides more than "trust", "belief", and endorsements. What we need are verifiable facts, validation, provable attribution, and confirmation.

I like small laptops. Years ago I got a 10-inch Asus EeePC with an Atom processor. It wasn't very powerful, but it ran Linux. Well, mostly. The audio drivers sometimes had problems and I never got Bluetooth to work. Battery storage capacity degrades over time. The EeePC battery originally lasted over 12 hours per charge, but after nearly a decade, it would get about 2 hours. I couldn't find a replacement battery, so five years ago I decided to get a new laptop.

The replacement laptop was a little larger (13-inch), but all of the components were supposed to be compatible with Linux. It also came with Windows 10 installed. I always intended to put Linux on it, but never got around to it. Since I only used it for web browsing and remote logins (using PuTTy), upgrading was never an urgency and Win10 was good enough.

However, over the years the laptop began to develop problems:

• The network sometimes wouldn't auto-connect after waking from suspend mode. I'd have to toggle the network on and off a few times before it would work. Other people had similar problems with Win10. Their solutions didn't work for me, but it wasn't annoying enough to replace the operating system.


• The mousepad's button began losing sensitivity. Just as I had worn through the keyboard on my desktop computer, I was wearing out the trackpad's button. But it wasn't bad enough to replace the entire laptop.


• With Win10 heading toward end-of-support (EoS is October 14, 2025), I knew I needed to upgrade the operating system sooner than later.

The final straw was the most recent Patch Tuesday. The laptop downloaded the updates, rebooted, and just sat at "Restarting". I couldn't figure out how to get past this. I'm sure there are instructions online somewhere, but I decided that it would be easier to install Linux.

(While I couldn't get back into the Windows system, I wasn't worried about backing up any files. This laptop is only used for remote access to the web and servers, and for giving presentations. All personal files already existed on my other systems.)

Intentional Procrastination

There's one reason I kept putting off installing Linux. It's not as simple as downloading the OS and installing it. (If that's all it took, I'd have done it years ago.) Rather, it usually takes a few days to customize it just the way I like it.

This time, I installed Ubuntu 24.04.2 (Noble Numbat). The hardest part was figuring out how to unlock the drive (UEFI secure boot). Otherwise, the installation was painless.

On the up side:

• The laptop is noticeably faster. (I had forgotten how much of a resource hog Win10 is.)


• The hard drive has a lot more room. (Win10 is a serious disk hog.)


• The network wakes up immediately from suspend. That was a Windows bug, and Linux handles it correctly.


• This is an older laptop. The battery originally lasted 8-9 hours under Windows, but had aged to lasting 4-6 hours from a full charge. With Linux, the same laptop and same old battery is getting closer to 10-12 hours, and that's while doing heavy computations and compiling code.


• Unexpected: The trackpad's buttons work fine under Linux. I thought I had worn out the physical contacts. Turns out, it was Win10.

On the downside, it's yet another Linux desktop, and that means learning new ways to customize it. (Linux is made by developers for developers, so the UI really lacks usability.)

Disabling Updates

My first customization was to disable updates. I know, this sounds completely backwards. However, I use my laptop when I'm traveling or giving presentations. I do not want anything updating on the laptop while I'm out of the office. I want the laptop to be as absolutely stable and reliable as possible. (I've seen way too many conference presentations that begin with the speaker apologizing for his computer deciding to update or failing to boot due to an auto-update.)

In the old days, there was just one process for doing updates. But today? There are lots of them, including apt, snap, and individual browsers.

• Snap: Snap accesses a remote repository and updates at least four times a day. (Seriously!) On my desktop computers, I've changed snap to update weekly. On my production servers and laptops, I completely disabled snap updates. Here are the commands to check and alter snap updates:
  
  
  • To see when it last ran and will next run: snap refresh --time --abs-time
  
  
  • To disable snap auto-updates: sudo snap refresh --hold
  
  
  • To restart auto-updating: sudo snap refresh --unhold
  
  
  • To manually check for updates: sudo snap refresh
  
  Now the laptop only updates snap applications when I want to do the update.


• Apt: In older versions of Linux, apt used cron to update. Today, it uses system timers. To see the current timers, use:
  

  systemctl list-timers --all

  Leave the housekeeping timers (anacron, e2scrub, etc.), but remove the auto-update timers. This requires using 'stop' to stop the current timer, 'disable' to prevent it from starting after the next boot, and optionally 'mask' to prevent anything else from turning it back on. For example:
  

  # Turn off apt's daily update.
sudo systemctl stop apt-daily-upgrade.timer
sudo systemctl disable apt-daily-upgrade.timer
sudo systemctl stop apt-daily.timer
sudo systemctl disable apt-daily.timer

# turn off motd; I don't use it.
sudo systemctl stop motd-news.timer
sudo systemctl disable motd-news.timer

  But wait! There's more! You also need to disable and remove some packages and settings:
  
  
  • Remove unintended upgrades: sudo apt remove unattended-upgrades
  
  
  • Edit /etc/apt/apt.conf.d/20auto-upgrades and set APT::Periodic::Update-Package-Lists and APT::Periodic::Unattended-Upgrade to "0".
  
  
  • And be sure to really disable it: sudo systemctl disable --now unattended-upgrades
  
  If you don't do all of these steps, then the system will still try to update daily.


• Ubuntu Advantage: Brian Krebs has his "3 Basic Rules for Online Safety". His third rule is "If you no longer need it, remove it." I have a more generalized corollary: "If you don't use it, remove it." (This is why I always try to remove bloatware from my devices.) Canonical provides Ubuntu Advantage as their commercial support, but I never use it. Following this rule for online safety, I disabled and removed it:
  

  sudo systemctl stop ua-messaging.timer
  
  sudo systemctl stop ua-messaging.service
  
  sudo systemctl stop ua-timer.timer
  
  sudo systemctl mask ua-messaging.timer
  
  sudo systemctl mask ua-messaging.service
  
  sudo systemctl mask ua-timer.timer
  
  sudo rm /etc/apt/apt.conf.d/20apt-esm-hook.conf
  
  sudo apt remove ubuntu-advantage-tools
  
  sudo apt autoremove



• Browsers: I use both Firefox and Chrome (Chromium). The problem is, both browsers often check for updates and install them immediately. Again, if I'm traveling or giving a presentation, then I do not want any updates.
  
  
  • I installed Chrome using snap. Disabling snap's auto-update fixed that problem. Now Chrome updates when I refresh snap.
  
  
  • Firefox was installed using apt. Disabling the browser's auto-update requires going into about:config. Search for "app.update.auto" and set it to "false". At any time, I can go to the browser's menu bar and select Help->About to manually trigger an update check.

While I turned off auto-updates, I set a calendar event to periodically remind me to manually perform updates on all of my computers. (I may not have the latest patch within hours of it being posted, but I do update more often than Window's monthly Patch Tuesday.) To update the system, either when the calendar reminds me or before going on a trip, I use:

sudo apt update ; sudo apt upgrade ; sudo snap refresh

Phone Home

I've configured my laptop, cellphone, and every other remote device to "phone home" each time they go online, change network addresses, or have a status update. One of my servers has a simple web service that listens for status updates and records them. This way, I know which device checked in, when, and from where (IP address). I also have the option to send back remote commands to the device. (Like "Beep like crazy because I misplaced you!") It's basically the poor-man's version of Apple's "Find My" service.

Figuring out where to put the phone-home script was the hard part. With Ubuntu 24.04, it goes in: /etc/network/if-up.d/phonehome. My basic script looks like this:

#!/bin/sh

curl 'https://myserver/my_url?status=Online' >/dev/null 2>&1

(Make sure to make it executable.) This way, whenever the laptop goes online, it pings my server. (My actual script is a little more complicated, because it also runs commands depending on the server's response.)

Desktop Background

I like a simple desktop. Few or no icons, a small task bar, and a plain dark-colored background. Unfortunately, Ubuntu has migrated away from having solid color backgrounds. Instead, Ubuntu 24.04 only has an option to use a picture. Fortunately, there are two commands that can disable the background picture and specify a solid color. (I like a dark blue.)

gsettings set org.gnome.desktop.background picture-uri none

gsettings set org.gnome.desktop.background primary-color '#236'

These changes take effect immediately.

Terminal Colors

With such as small laptop screen, I don't want large title bars or borders around windows. However, the Ubuntu developers seem to have taken this to an extreme. I spend a lot of time using the command-line with lots of terminal windows open. The default terminal has a dark purple background (a good, solid color) and no visible border around the window. But that's a problem: If I have three terminal windows open, then there is almost no visual cue about where one terminal window ends and the next begins.



I quickly found myself constantly fiddling with title bars to figure out which terminal window was on top and wiggling the window's position to figure out where the borders were located. Even with tabbed terminal windows, there is very little visual distinction telling me which tab is active or letting me know when I've switched tabs.

After a few days of this, I came up with a workaround: I give every terminal window a different background color. Now there's a clear visual cue telling me which window and tab is active.



The default shell uses bash, which means it runs $HOME/.bash_aliases each time a new window is opened. Here's the code I added to the end of my .bash_aliases file:

##### Set terminal background color based on terminal number

# get terminal name, like: /dev/pts/0

termnum=$(tty)

# reduce the name to the number: /dev/pts/1 become 1

termnum=${termnum##*/}

# I have 10 unique colors; if more than 10 terminals, then repeat colors

((termnum=$termnum % 10))

# set the color based on the terminal number, using escape codes.

case $termnum in

  0) echo -n -e "\e]11;#002\e\\" ;;

  1) echo -n -e "\e]11;#010\e\\" ;;

  2) echo -n -e "\e]11;#200\e\\" ;;

  3) echo -n -e "\e]11;#202\e\\" ;;

  4) echo -n -e "\e]11;#111\e\\" ;;

  5) echo -n -e "\e]11;#220\e\\" ;;

  7) echo -n -e "\e]11;#321\e\\" ;;

  8) echo -n -e "\e]11;#231\e\\" ;;

  9) echo -n -e "\e]11;#123\e\\" ;;

esac

Now I can have five open terminals, each with a different background color. Each terminal is easy to distinguish from any adjacent or overlapping windows.

Almost Done

It took about two hours to install the laptop. (That includes downloading Ubuntu, copying it to a thumb drive, and installing it on the laptop.) Many of my customizations, such as setting up my remote server access and setting my preferred shell preferences, were straightforward.

Switching from Windows to Ubuntu gave this older laptop a lot of new life. But with any new system, there are always little things that can be improved based on your own preferences. Each time I use the laptop, I watch for the next annoyance and try to address it. I suspect that I'll stop fiddling with configurations after a month. Until then, this is a great exercise for real-time problem solving, while forcing me to dive deeper into this new Ubuntu version.

I recently traveled to see some extended family and friends. Of course, that means PTSD (People Tech Support Duties). I did everything from hardware (installing a grab bar in the shower) to software (showing how to use AirDrop to transfer a picture -- because there's no way I'm plugging my personal device into their equipment).

Having said that, I did receive one really odd question, but it needs some context. An older friend constantly watches FOX News, spends way too much online time in far-right extremist forums on Facebook, and isn't very tech savvy. His question: How does he get a burner phone?

Huh? What? Why are they talking about burner phones in those far-right forums? And why does he think he needs a burner phone? Keep in mind, he currently doesn't have a cellphone at all. Perhaps it's time to set the story straight about burner phones and how they, by themselves, really don't help with anonymity.

Go for the Burn

Let's start at the beginning: a "burner phone" is not a brand or style of phone. Any cellphone can be a burner phone. "Burner" just means that there's no impact to you if it is lost or discarded.

Keep in mind, some people don't use a cellphone and only want it when traveling. (This is my friend's situation.) If you just want a short-term phone with no long-term contract, then you don't need a burner. Instead:

• Get a pre-paid phone. When you need it, you can add minutes and use the phone. And if you don't need it for months at a time, then just store it with your other travel supplies.


• Use a service like Consumer Cellular, Cricket, Visible, Tracfone, or T-Mobile's Metro. (Not an endorsement; just examples.) They don't have long-term contracts and you can cancel or suspend the service when you want. This way, you don't have to pay for a monthly service if you don't need it that month.

In either case, you can pay one time for a cheap phone, then put on minutes or enable the service as needed. This isn't a burner phone; this is a travel phone or a no-contract phone.

Burner Level 1: Phone Number

Cellphones have a bunch of identifiers used by phone companies to track the service. However, they can also be used by law enforcement (or "hackers") to track people. This is why it's important to identify why you want a burner and what you consider to be your threat.

At the most superficial level is your phone number. People with this number can contact you. A burner number is a phone number that you can easily change whenever you want. This is often useful if you have an ex-partner or ex-employer who you no longer want to talk to. You can change your number, inform your friends, and exclude the people you want to avoid.

Of course, this can be a hardship if you use your phone number for work or have hundreds of friends. There are also some risks associated with changing your phone number:

• If you have lots of friends, one of them could leak your new number to the person you're trying to avoid. (This is a real problem when your crazy ex-partner has a lot of common friends.)


• If you get a new phone number, then you inherit all of the crazy people who were ghosted by the former owner of that same number. Remember: who changes phone number? People with crazy stalkers and debt collectors.


• Many online services use your phone number for two-factor authentication. If you lose your number, you lose your 2FA. Be sure to disable 2FA on every service before getting rid of the phone, and enable 2FA on your new phone number. However, some services require 2FA. In that case, get the new number and transfer all of your 2FA registrations before ghosting your old number. (This is a painful process.)


• If you stay with the same carrier, then changing numbers may include a "call forwarding" service. Be sure to disable that, otherwise your crazy stalkers will just follow you to your new number.

There are some better alternatives to changing phone numbers. For example, I never give out my direct cell number. Instead, I use Google Voice (GV). GV will ring my office, cellphone, and any other numbers I use. I can easily change my cellphone number without anyone knowing. GV also provides filtering services, like spam call blocking and "state your name" before forwarding the call to me. This is a better approach for filtering out people you no longer want to talk to.

As another option, some carriers permit you to change numbers a few times each year. This is useful if you regularly use burner numbers.

Burner phone numbers are for ghosting known people who you no longer want to communicate with. However, they won't stop law enforcement from tracking you. A simple warrant to the carrier is often enough to get your old and new numbers.

Burner Level 2: SIM

The Subscriber Identity Module (SIM) is a small smartcard that links the phone to your account at the phone carrier. It contains the International Mobile Subscriber Identity (IMSI), which is a unique number for tracking the account, an authentication key (to deter impersonating someone else's account), your phone's PIN code, mobile country code and mobile network code (MCC and MNC), and a few other pieces of unique information. The SIM also includes a little storage area (like 8KB or 256KB), but you're not going to store hundreds of contacts or camera photos in the SIM.



You can change phone numbers without changing the SIM, because the carrier assigns that. You can also change phones without changing phone numbers by moving the SIM into the new device. (You might also need to tell your carrier that you are using a new device, since the carrier needs both the SIM and device information. However, some carriers auto-detect the new device when they see the IMSI associated with a new phone.)

Since the SIM is tied to your account, a burner account would use a different SIM that isn't associated with your normal billing account at the carrier. Some carriers offer pre-paid phone services, so the SIM is relatively anonymized -- it doesn't need to be tied to a person since the account has funds already paid.

When you get rid of your phone, be sure to remove the SIM. Either move it to the new phone, or destroy it. (Use scissors to cut it into tiny pieces; be sure to crack the chip inside it.)

Burner Level 3: IMEI

Every cellphone has a unique International Mobile Equipment Identity (IMEI). Let's say you give your phone to a friend as a hand-me-down. You remove your SIM and your friend inserts their SIM. Somewhere among the phone carriers is a record that the physical device (IMEI) had been associated with your IMSI and phone number, and now is associated with your friend's IMSI and phone number.

A burner phone means you want a new device with a new IMEI. However: If you moved your SIM to the new phone, then the IMEI changes but the IMSI stays the same. This means that someone with a warrant can track you. If you want a burner phone, then use a new device (IMEI), new SIM (IMSI), and new phone number. Everything needs to be new.

If you want to get rid of your old phone, then:

1. Copy off anything you want to keep. (Photos, contacts, etc.)


2. Remove the SIM and any additional storage media (SDcard).


3. Perform a factory reset. (How to do this depends on the phone.) This will remove your call logs, photos, contacts, and everything else.


4. If you're super paranoid like me, then destroy the phone. (Open it up. If it looks like a memory chip, then drill through it.)

In TV shows and movies, you often see someone discard a burner phone by stomping on it. However, while it makes for good cinema, you're probably just going to crack the screen. The memory, SDcard, and SIM are probably still intact. (Inside the phone is a lithium battery. If you don't step hard enough to crack the battery, then the memory is probably recoverable. And if you do crack the battery, then you have seconds to get clear before it catches fire and possibly explodes.)



The better way to dispose of a burner phone: Perform a factory reset to remove the call logs and contacts, then "accidentally" leave the phone in a crowded area. Someone else is almost certain to find it and walk off with it. This way, anyone who is tracking the IMEI and IMSI will start following the wrong person who's heading in a different direction.

Burner Level 4: Anonymity

Just getting a burner phone is not enough to be anonymous. You also need to consider operational security (OPSEC).

Consider this: Who are you going to call with your burner phone? If you call anyone who you previously called with the old phone, then there's an immediate trail from that known number back to your new phone.

By the same means, where are you going to go online? You might have a new IMEI, new IMSI, and new phone number, but if you log into your existing Facebook account, then you just linked everything back to the new no-longer-anonymous phone. Even if you create a new Facebook account, if you reach out to any of your previous contacts or forums, then you just linked back to yourself.

But it's actually worse than that. Often in movies, you'll see a gang of thieves or spies distribute new burner phones before starting a caper. They all receive their new phones and then immediately check that the phones work. The problem is, cell towers triangulate phone locations. (This is how the phone knows which tower has the strongest signal to use.) If a bunch of new phones turn up at the same physical location, then they can immediately be linked together. If law enforcement recovers one phone, then they can trace the cell tower records back to a location where there were other new phones. Then the feds can guestimate how many people are in the group (assume one per phone) and track where they are now.

To really be anonymous, you need new phones with new SIMS and new phone numbers that only call other new numbers and use new accounts that were never linked back to any old identities. And you need to do this without getting any of the new devices into the same vicinity as any other new device. (This is much harder than it sounds.) Like the T-800 said to Sarah Connor in Terminator: Dark Fate: "If you're going to keep your phone in a bag of potato chips, then keep your phone in a bag of potato chips." It just takes one mistake to link any tracking back to you and all of your other co-conspirators.

Burner Level 5: Naked Man

At the far extreme end is a spy technique that is informally called the "naked man" (not literally naked). The idea is that an operative arrives in a new location with nothing: no phone, no laptop, no technology. When they need a phone or laptop, they buy it there (using cash). If they need an online account, they make one as they need it. This creates a new online identity with no links back to their previous identity.

There's just one problem: everyone leave a digital trail. Having someone suddenly appear on the grid without any previous trail is a red flag and stands out like a sore thumb. Consider this:

• How many people go through an airport with zero electronics? You stand out as a naked man. (Well, unless you're really old or traveling from a third-world nation. But between developed nations? Tech is the norm.)


• Okay, so to look like everyone else, you enter the airport with a burner and you ditch it when you arrive. Except that now there's a record of the previous phone going offline and a new entity with zero digital trail coming online. The authorities can often rule out everyone who has an established digital trail and then identify you.


• You go to buy new tech using cash. Due to theft and robbery, nearly all stores have cameras at the registers. Since the cameras capture your face, you're no longer anonymous, and since you're paying with cash, you stand out. In addition, some stores won't accept lots of cash for a sale; having too much cash at the register makes them a big target for armed robbery. (There's a known scam that hits retailers: buy something expensive with lots of cash, then have your friend rob the store to recover the cash.) Big transactions usually require a credit card, but a credit card will link the transaction back to you. (Even pre-paid credit cards are not anonymous, but that's a topic for some other blog entry.)


• And we get back to the same burner problem: You have your newly purchased tech. How are you going to contact your remote associates? The instant that you reach out to anyone, you create a digital link back to yourself.

Practicing good OPSEC and tradecraft isn't something you can learn from a blog. This takes serious training, consideration, and support infrastructure. The dark conspiracy groups on Facebook who are talking about using burner phones? Those are the rank amateurs who are certain to get caught.

I'm continuing to look for ways to lower my energy bill, even if only by a few dollars. One of my ideas was to use solar panels. However, the roof on the office building isn't ideal for solar.

• The optimal direction is East to South-East for morning and South-West to West for afternoon. Unfortunately, the southern facing parts of the roof have lots of small sections, so there's no place to mount a lot of solar panels. But I do have space for a few panels on the roof; probably enough to power the server rack.


• All of the professional solar installation companies either don't want to install panels if it's less than 100% of your energy needs, or they want to charge so much that it won't be worth the installation costs. This rules out the "few solar panels" option from a professional installer.

Last year, I decided that it would be a good learning experience to make my own solar panel Energy Storage System (ESS). My goal was not to power the entire office or sell power back to the electric company. Rather, I wanted an off-grid solution to just power the server rack for a few hours each day. If it worked, it should save me somewhere between 20kWh and 40kWh per month. That's less than 10% of my utility bill, but it's better than nothing. And assuming I ran the numbers correctly, it should pay itself off in about 5 years. (I hoped to keep the costs significantly lower by doing the installation by myself.)

In the worst case, it may never earn enough to pay itself off. But at least I'll learn something about solar panels, energy storage systems, and high voltage.

Having said that, developing it myself was certainly full of unexpected surprises and learning curves. Each time I thought I had everything I needed, I ended up finding another problem. (Now I know why professional installers charge tens of thousands of dollars. I don't even want to think about how much of my labor that went into this.)

The Basic Idea

I started this project with a basic concept. For the rest of the details, I decided that I'd figure it out as I went along.

1. Goal: I want an off-grid solar powered system for my server rack. It is not intended to run 24 hours a day, cover all of my energy needs, or sell excess power back to the utilities. I only want to reduce my power usage and related costs by a little. (When I consulted with professional solar installers, this is a concept that they could not comprehend.)


2. Low budget: A professional installation can cost over $20,000. I want to keep it under $1,000. For me, I wanted this to be a learning experience that included solar power and embedded controllers.


3. Roof: The original plan was to put some panels on the roof. Since I don't have much roof space, I was only going to have two panels that, under ideal conditions, could generate about 100 watts of power each, forming a 200W solar system. This won't power the entire office, but it should power the server rack for a few hours each day. (I ended up not going with a roof solution, but I'll cover that in a moment.)

The entire architecture is kind of overwhelming. Here's a drawing that shows what I'm doing:



And here's the final system:



Note: I'm naming a lot of brands to denote what I finally went with. This is not an endorsement or sponsorship; this is what (eventually) worked for me. I'm sure there are other alternatives, and I didn't necessarily choose the least expensive route. (This was a learning experience.)

• Solar charger: The solar panels connect to a battery charger, or Maximum Power Point Tracking (MPPT) system. The MPPT receives power from the solar cells and optimally charges the battery. Make sure to get an MPPT that can handle all of the power from your panels! My MPPT is a Renogy Rover 20, a 20-amp charger that can handle a wide range of batteries. The two black wires coming out the bottom go to the battery. There's also a thin black line that monitors the battery's temperature, preventing overcharging and heat-related problems. Coming off the left side are two additional black lines that connect to the solar panels. (The vendor only included black cables. I marked one with red electrical tape so I could track which one carried the positive charge.) There's also a 10-amp fuse (not pictured) from the solar panels to the MPPT.


• Battery: The MTTP receives power form the panels and charges up a moderately large battery: 12V 100Ah LiFePO4 deep cycle battery. (Not pictured; it's in the cabinet.) When fully charged, the battery should be able to keep the servers running for about 30 minutes.


• Inverter: On the right is a Renogy 2000W power inverter. It converts the 12V DC battery into 120V 60Hz AC power. It has two thick cables that go to the battery, with red going through a 20-amp fuse. (Always put fuses on the red/positive lines.)


• Automatic Transfer Switch (ATS): At the top (yellow box) is the automatic transfer switch (ATS) that toggles between utility/wall power and the inverter's power. It has a 30ms transfer speed. I had been using this box for years to manually switch between power sources without interruption. The three cables coming out of it go to the two inputs: primary is the inverter and fallback is the wall outlet. The output line goes to the UPS in the server rack. The UPS ensures that there isn't an outage during the power transfer. It also includes a power smoother to resolve any potential power spikes or phase issues.


• Output power: The ATS's output AC power (from grid or inverter) goes into a smart outlet (not pictured in the line drawing, but visible in the photo below as a white box plugged into the yellow connector at the top). This allows me to measure how much power the server rack consumes. There's a second smart outlet (not pictured) between the wall outlet and the ATS, allowing me to measure the power consumption from the utility. When I'm running off grid power, both smart outlets report the same power consumption (+/- a few milliamps). But when I'm running off the inverter, the grid usage drops to zero.


• Controller: In the middle (with the pretty lights) is my DIY embedded controller. It reads the battery level and charging state from the MPPT and has a line that can remotely turn on and off the inverter. It decides when the inverter runs based on the battery charge level and available voltage. It also has a web interface so I can query the current status, adjust parameters, and manually override when it runs.


• Ground: Not seen in the picture, there's grounding wire from the inverter's external ground screw to the server rack. The server rack is tied to the building's "earth ground". Proper grounding is essential for safety.

Everything is mounted vertically to a board that is hung from the side of the server rack. This allows me to easily take it down for any maintenance issues. (And when doing the initial testing, I could carry the entire thing outside.)

Even though I knew I'd be starting this project around March of this year, I started ordering supplies five months earlier (last November). This included solar panels, a solar charger, battery, and an inverter. I ordered other components as I realized I needed them. Why did I start this so early? I believed Trump when he said he would be imposing stiff tariffs, making everything more expensive. (In hindsight, this was a great decision. If I started ordering everything today, some items would cost nearly twice as much.)

Measuring Power

Before starting this project, I needed to understand how much power I'd require and how much it might save me on my utility bill.

As a software (not hardware) person, I'm definitely not an electrical engineer. For you non-electricians, there are three parts of electricity that need to be tracked:

• Voltage (V). This is the amount of power supplied on the wires. Think of it like the pressure in a water pipe.


• Amps (A). This is the amount of current available. Think of this like the size of the water pipe. A typical desktop computer may require a few amps of power. Your refrigerator probably uses around 20 amps when the compressor is running, while an IoT embedded device usually uses 200mA (milliamps, or 0.2A, that's flea power).


• Watts (W). This is the amount of work available. W=A×V.

These measurements are often compared to flowing water. Volts identify how fast a river is flowing (the water pressure). Amps identify how large the river is, and watts is the total energy delivered by the river. A wide but slow moving river has high amps but low voltage. A narrow but fast flowing river has a low current (low amps) but a high voltage. Because of the relationship between W, A, and V, the electronics can adjust the A and V distribution while mantaining the same W.

W, A, and V are instantaneous values. To measure over time, you typically see Watt-hours (Wh) and Amp-hours (Ah). Your utility bill usually specifies how many Wh you used (or kilowatts for 1000 Wh; kWh), while your battery will identify the amount of power it can store in terms of Ah at a given V. If you use fewer amps, then the battery will last longer.

Keep in mind, this can really screw up the power calculations if you get them wrong. For example, my 12V 100Ah DC battery is being converted to 120V AC power. If the AC uses a 1-amp load (like one server in the rack), then that's not 100 hours of battery; that's 10 hours. Why? 12V at 100Ah is 1200Wh. 1200Wh&div;120V=10Ah, or 10 hours of power. (And with inverter's overhead and conversion loss, it's actually less.)

Parts and Parts

While I work with computers daily, I'm really a "software" specialist. Besides a few embedded systems, I don't do much with hardware. Moreover, the computer components that I deal with are typically low voltage DC (3.3V, 5V, or 12V and milliamps of power; it's hard to kill yourself if you briefly short out a 9V battery).

When it comes to high voltage, my electrical engineering friends all had the same advice:

1. Don't kill yourself.


2. Assume that all wires have enough power to kill you. Even when turned off.


3. When possible, over-spec the components. If you need 5 amps, get something that can handle 10 amps. If you need 12 gauge wire (12awg), then use 8awg (a thicker wire). If you need 2 hours of power, get something that can provide 4 hours of power. You can never go wrong by over-spec'ing the components. (Not exactly true, but it's a really good heuristic.)

For the last year, I've been using some Shelly plugs to monitor the energy consumption of my server rack. Every hour I take a reading and store it in a database. I also wrote a web interface that can display the real-time information and graph the hourly usage. (Every vertical bar is an hour, and every color is one day.)



The lower part of the rack hosts FotoForensics, Hacker Factor, and my other primary services. It usually consumes about 230W of power, or 2A. (It can fluctuate up to almost 300W during a reboot or high load, but those don't last long.) The upper rack is for the development systems, and uses around 180W. (180W at 120V is 1.5A.) That's right, the entire rack is usually consuming less than 4Ah of power at any given time.

For this solar experiment, I decided to initially only power the upper rack with solar. (If it turns out to be really successful, then I might add in the lower rack's power needs.)

The Bad Experiences

I had a few bad experiences while getting this to work. I chalk all of them up to the learning curve.

Problem #1: The Battery
Setting up the MPPT, inverter, and ATS was easy. The battery, on the other hand, was problematic. There are lots of batteries available and the prices range wildly. I went with a LiFePO4 "deep cycle" battery because they last longer than typical lead acid and lithium batteries and are designed for repeatedly powering up and draining. LiFePO4 also doesn't have the "toxic fumes" or "runaway heat" problems that the other batteries often have.

I found a LiFePO4 battery on Amazon that said it was UL-1973 certified. (That means for use with a solar project.) However when it arrived, it didn't say "UL 1973" anywhere on the battery or manuals. I then checked with Underwriter Labs web site. The battery was not listed. The model was not listed. The brand was listed, but none of their products had UL certifications. This is a knock-off forgery of a battery. If they lied about their certification, then I'm not going to trust the battery.

Amazon said that the vendor handles returns directly. My first request to the vendor was answered quickly with an unrelated response. I wrote to them: "I'd like to return the battery since it is not UL certified, as stated on your product description page." The reply? "The bluetooth battery needs to be charged before you can use it." (This battery doesn't even have bluetooth!)

My second request to the vendor received no response at all.

I told my credit card company. They stopped payment, sent an inquiry to the vendor, and gave them 15 days to respond. Two weeks later, with no response, I was refunded the costs. The day after the credit card issued the refund, the vendor reached out to me. After a short exchange, they paid to have the battery returned to them.

The second battery that I ordered, from a different vendor, had all of the certificates that they claimed.

Problem #2: The Inverter
The first inverter that I got looked right. However, when I connected it to the ATS, the wall outlet's circuit breaker immediately tripped. Okay, that's really bad. (But also, really good that the circuit breaker did its job and I didn't die.) It turns out, inverters above a certain wattage are required to have a "neutral-ground bond". The typical American three-prong outlet has a hot, neutral, and ground wire. The N-G bond means that neutral and ground are tied together. This is a required safety feature. Every home and office circuit has exactly one N-G bond. (It's in the home or building's circuit breaker panel.)

The four-poll (4P) ATS ties all grounds together while it switches the hot and neutrals. The problem: If the inverter and wall outlet both have a N-G bond, then it creates a grounding loop. (That's bad and immediately trips the circuit breaker.) For most inverters, this functionality is either not documented or poorly documented. Some inverters have a built-in N-G bond, some have a floating neutral (no bond) and are expected to be used with an ATS, and some have a switch to enable/disable the N-G bond.

My first inverter didn't mention the N-G bond and it couldn't be disabled. Fortunately, I was able to replace it with one that has a switch. With the N-G bond safely disabled, I can use it with the ATS without tripping the circuit breaker.

Keep this in mind when looking for an inverter. Most of the ones I looked at don't mention how they are bonded (or unbonded).

Problem #3: The ATS
I spent days tracking down this problem. The ATS output goes to a big UPS. This way, any transfer delays or phase issues are cleaned up before reaching the computers. When the inverter turned on, I would see a variety of different problems:

• Sometimes the UPS would run fine.


• Sometimes the UPS would scream about an input problem, but still run off the input power.


• Sometimes the UPS would not scream, and would slowly drain its internal battery while also using the inverter's power.


• Sometimes the UPS would scream and refuse to use the input power, preferring to run off the UPS battery.

The problem was incredibly inconsistent.

If I removed the ATS, then the UPS had no problem running off utility power. If I moved the electrical plug manually to the inverter (with the N-G bond enabled), it also ran without any problems.

Long story short: Most automatic transfer switches have a "direction". If primary is utility and backup is the generator (or solar), then it demands to be installed in that direction. For my configuration, I want a battery-priority ATS, but most ATSs (including mine) are utility-priority. You cannot just swap the inputs to the ATS and have it work. If, like me, you swap them, then the results become incredibly inconsistent and will lead you down the wrong debugging path.

My solution? Someday I'll purchase a smart switch that is battery-priority. In the meantime, I have a Shelly smart-plug monitoring the utility power. My DIY smart controller tells the Shelly plug to turn on or off utility power. When it turns off, the ATS immediately switches over to using the inverter's power. And when my DIY controller see that the solar battery is getting low, it turns the utility grid back on.

The added benefit for my method of turning on or off the utility power is that I can control the switching delay. The inverter takes a few seconds to start up. I have a 15-second timer between turning on the inverter (letting it power up and normalize) and turning off the utility power. This seems to help the UPS accept the transfer faster.

Problem #4: Over-spec'd Inverter
Remember that advice I got? Always over-spec the equipment? Well, that's not always a good idea. As it turns out, a bigger inverter requires more energy to run (18Wh for a 2000W inverter vs 12Wh for a 1000W inverter). It also has a worse conversion rate for a low load. (The inverter claims to have >92% conversion rate, meaning that the 100Ah battery should last for 92Ah. But with a light load, it may be closer to 80%.)

I'll stick with the inverter that I got, but I could probably have used the next smaller model.

Problem #5: The Roof
I wanted to put the solar panels on the roof. I really thought this was going to be the easiest part. Boy, was I wrong.

There are federal, municipal, and local building requirements, and that means getting a permit. The city requires a formal report from a licensed structural engineer to testify that the roof can hold the solar panels. Keep in mind, I'm talking about two panels that weigh 14lbs (6kg) each. The inspector who goes up on the roof weighs more. If a big bird lands on my roof (we have huge Canadian geese), it weighs more. We get snow in the winter and the snow weighs more!

Unfortunately, the city made it clear that there is no waiver. I had earmarked $1000 for everything, from the panels to the battery, inverter, wires, fuses, mounting brackets, etc. I got quotes from multiple structural engineers -- they all wanted around $500. (There goes my budget!) And that's before paying for the permit. In effect, the project was no longer financially viable.

Fortunately, I found a workaround: awnings. The city says that you don't need a permit for awnings if they (1) are attached to an exterior wall, (2) require no external support, and (3) stick out less than 54 inches. My solar panels are mounted at an angle and act as an awning that sticks out 16 inches. (The mounts are so sturdy that I think they can hold my body weight.) No permit needed.

The awnings turned out to be great! They receive direct sunlight starting an hour after sunrise and it lasts until about 1pm in the summer. (It should get even more in the winter.) They continue generating power from ambient lighting until an hour before sundown. This is as good as having them on the roof!

The Scariest Part

High voltage scares me. (That's probably a healthy fear.) Connecting cables to the powered-off system doesn't bother me. But connecting wires to the big battery is dangerous.

Using rubber-gripped tools, I attached one cable. However, when I tried to connect the other cable, there was a big spark. It's a 100Ah battery, so that's expected. But it still scared the donuts out of me! I stopped all work and ordered some rubber electrical gloves. (Get rubber or nitrile, and make sure they are class 00 or higher.)

Along with the gloves, I ordered a huge on/off switch. This isn't your typical light switch. This monster can handle 24V at 275 amps! (It's good to over-spec.)

I connected the MPPT and inverter to one side of the switch. An 8awg cable that can handle 50 amps connects to the battery's negative pole. (Since the MPPT and inverter are both limited to 20 amps, the 50 amp cable shouldn't be a problem.)

With the gloves on, the switch powered off, and rubber-gripped tools in hand, I connected the switch to the battery. No zap or spark at all. Turning the switch on is easy and there is no spark or pop. This is the right way to do it.

Expected Savings

Without the solar project (just using the utility power), the server rack costs me about $35 per month in electricity to run.

I've been running some tests on the solar project's performance, and am very happy with the results.

Under theoretically ideal conditions, two 100W panels should generate a maximum of 200W. However, between power conversion loss, loss from cabling, and other factors, this theoretical maximum never happens. I was told to be happy if it generated a maximum of 150W. Well, I'm very happy because I've measured daily maximums between 170W and 180W received at the MPPT.

Fort Collins gets over 300 sunny days a year, so clear skies are the norm. With clear skies, the battery starts charging about 30 minutes after sunrise and gets direct (optimal) sunlight between 9am and 1pm. It generates an incredible amount of power -- the inverter drains the battery slower than the panels can charge it. For the rest of the afternoon, it slowly charges up the battery through indirect ambient light. The net result? It can run the upper half of the rack for over 10 hours.

This kind of makes sense:

• The battery usually starts the day at about 50% capacity. It charges to 90% in under 2 hours of direct daylight.


• In theory, I run the battery from 90% down to 20%. In practice, the battery usually hits 100% charged during the morning because if charges faster than it drains. It doesn't start running below 100% until the afternoon. (That's wasted power! I need to turn on more computers!)


• I'm only using the upper rack right now (1.5Ah, 180Wh). The inverter consumes another 18W, so call it 200W of power. Assume a fully-charged 1200Wh battery with 920Wh available, draining at a rate of 200Wh. It should last about 4.5 hours. If I power the entire rack, it will be closer to 2 hours. And in either case, that countdown only starts when it's running off of battery in the late afternoon.

We had one dark and stormy day, and one very overcast day so far. In both instances, it took most of the morning to charge the battery, but it still managed to run the upper rack for a few hours. Fort Collins has "surge pricing" for electricity. In the summer, than means 2pm to 7pm has the most expensive power (about 3x more than non-surge times). Fortunately, the batteries keep the rack running during much of that expensive period.

I'm aiming to use the battery during the surge pricing period. If I ran the numbers correctly, the server might reduce my $35/month cost by $20-$25/month. At that rate, it will pay off the $1000 investment in under 4.5 years. If we have a lot of bad weather, then it might end up being 5 years. The batteries and panels will need to be replaced in 8-10 years, so as long as it pays off before then, I'll be in the profit range.

As self-paced learning goes, I don't recommend high voltage as an introductory project. Having said that, I really feel like I've learned a lot from this experiment. And who knows? Maybe next time I'll try wind power. Fort Collins has lots of windy days!

Today's online environment permits the easy manipulation and distribution of digital media, including images, videos, and audio files, which can lead to the spread of misinformation and erode trust. The media authentication problem refers to the challenge of verifying the authenticity and integrity of digital media.

For four years, I have been reviewing the solution offered by the Coalition for Content Provenance and Authenticity (C2PA). In each blog entry, I either focused on different types of exploits, exposed new vulnerabilities, or debunked demonstrations. You would think that, after nearly 30 blogs, that I would run out of new problems with it. And yet...

I'm not the only person evaluating C2PA. I know of three other organizations, four government groups, and a half dozen companies that are doing their own independent evaluations. I learned about these groups the easy way: they start with an online search, looking for issues with C2PA's solution, and Google returns my blog entries in the first few results. They reach out to me, we chat, and then they go on with their own investigations. (With my blog entries, it's not just me voicing an opinion. I include technical explanations and step-by-step demonstrations. It's hard to argue with a working exploit.) In every case, they already had their own concerns and suspicions, but nothing concrete. My working examples and detailed descriptions helped them solidify their concerns.

"I've got a little list."

Near the beginning of this year, a couple of different groups asked me if I had a list of the known issues with C2PA. At the time, I didn't. I have my own private notes and my various blog entries, but no formal list. I ended up going through everything and making a bulleted document of concerns, issues, and vulnerabilities. The initial list was 15 PAGES! That's 15 pages of bulleted items, with each bullet describing a different problem.

I had the paper reviewed by some technical peers. Based on their feedback, I changed some statements, elaborated on some details, added more exploits, and included lots of references. It grew to 22 pages.

I noticed that Google Docs defaulted to really wide margins. (I'm not sure why.) I shrank the margins to something reasonable. The reformatting made it a 20-page paper. Then I added more items that had been overlooked. Today, the paper is 27 pages long, including a one-page introduction, one-page conclusion, and some screenshots. (The bulleted list alone is about 24 pages.)

With blogs, you might not realize how much information is included over time. It wasn't until I saw over 20 pages of bulleted concerns -- strictly from public information in my blogs -- that I realized why I was so passionate about these concerns. It's not just one thing. It's issue after issue after issue!

The current paper is almost entirely a list of items previously mentioned in my blog. There are only a few bullets that are from my "future blog topics" list. (After four years, you'd think I'd run out of new concerns. Yet, I'm still not done. At what point should people realize that C2PA is Adobe's Edsel?)

Last week, C2PA, CAI, and IPTC held their "Content Authenticity Summit". Following this closed-door conference, I was contacted by some folks and asked if they could get a copy of my "C2PA issues" list. I decided that, if it's going to get a wider distribution, then I might as well make my paper public. Having said that, here's a link to the current paper:

C2PA Issues and Evaluation (public release, 2025-06-09)
If you have questions, comments, thoughts, think there is something that should be added, or see any errors, please let me know! I'm very open to updating the document based on feedback.

Organizing Content

There are many ways to organize this data, such as by risk level, functionality, technical vs. usability, etc. I decided to organize the document by root cause:

• Fundamental problems: Issues that cannot be fixed without a significant redesign.


• Poor design decisions: Issues that could be fixed if C2PA changed their approach. (However, C2PA has been very clear that they do not want to change these decisions.)


• Implementation issues: Bugs that could be fixed. (Many haven’t been fixed in years, but they are fixable with little or no undesirable side-effects.)

There are also topics that I am intentionally not including in the paper. These include:

• Severity. While I don't think there are many minor issues, I don't try to classify the severity. Why? Well, one company might think an issue is "critical" while another may argue that it is "severe" or "medium". Severity is subjective, based on your requirements. I think the paper provides enough detail for readers to make their own judgment call.


• Simple bugs. The existing code base is far from perfect, but I really wanted to focus only on the bugs due to the specifications or design decisions. Problems due to the specifications or policies impact every implementation and not one specific code base. However, the paper does include some complicated bugs, where addressing them will require a significant effort. (For example, c2patool has over 500 dependency packages, many of which are unvetted, include known vulnerabilities, and have not been patched in years.)


• Solutions. With few exceptions, I do not recommend possible solutions since I am in no position to fix their problems. There may be an undisclosed reason why C2PA, CAI, or Adobe does not want to implement a particular solution. The paper does include some solution discussions, when every solution I could think of just introduces more problems; sometimes developers work themselves into a corner, where there is no good solution.


• Alternatives. Except as limited examples, I intentionally avoid discussing alternative technologies. (And yes, there are some alternatives with their own trade-offs.) I want this paper to only be focused on C2PA.


• Wider ramifications. If C2PA is deployed in its current state, it can lead to incredibly serious problems. It would become the start of something with a massive domino effect. Rather than focusing on theoretical outcomes, the paper is directed toward immediate problems and direct effects.

Simply seeing a random list of problems can be overwhelming. I hope this type of organization makes it easier to absorb the information.

The Most Important Issue

While the first question I usually receive is "Do you have a list?", the second question is almost always "What is the most important issue?" Looking at the bulleted list, you might think it would be the misuse of X.509 for certificate management. I mean, lots of problems in the paper fall under that general topic. However, I think the bigger issue is the lack of validation. C2PA is completely based on 'trust':

• You trust that the name in the certificate represents the signer.


• You trust that the information in the file (visual content, metadata, timestamps, etc.) is legitimate and not misrepresented by the signer.


• You trust that the signer didn't alter any information related to the dependencies.


• You trust that the timestamps are accurate.

And the list goes on. C2PA provides authentication without validation and assumes that the signer is not intentionally malicious. However, if I trust the source of the media, then I don't need C2PA to tell me that I can trust it. And if I don't trust the signer, then nothing in C2PA helps increase the trustworthiness.

Alternatives

Following some of my initial critiques on my blog, representatives from C2PA and CAI (all Adobe employees) asked me for a video chat. (This was on December 13, 2023.) During the call, C2PA's chief architect became frustrated with my criticisms. He asked me if I could do better. A few weeks later, I created VIDA, which was later renamed to SEAL: the Secure Evidence Attribution Label. SEAL is still being actively developed, with some great additions coming soon. The additions include support for derivation references (simple provenance), support for offline cryptographic validation, and maybe even support for folks who don't have their own domains.

SEAL is a much simpler solution compared to C2PA. While C2PA tries to do a lot, it fails to do any of it properly. In contrast, SEAL focuses on one thing, and it does it incredibly well.

Just as I've been critical of C2PA, I've been looking at SEAL with the same critical view. (I want to know the problems!) I've had my technical peers also review SEAL. I instructed them to be brutal and hyper-critical. The result? Two pages of bullets (it's a 3 page PDF with an introduction). Moreover, almost every bullet is from the general problem of relying on DNS and networked time servers. (Everyone with a domain or time-based signing service has these problems; it's not just SEAL. If/when someone solves these problems, they will be solved for everyone, including SEAL.)

While I don't think the C2PA paper has many "minor" issues, I think all of SEAL's issues appear to be minor. For example, problems like domain squatting apply to anyone with a domain. It doesn't directly impact your own domain name, but could fool users who don't look too closely.

Here's the current draft of the SEAL Issues (2025-06-09 draft). Again, if you see anything wrong, missing, or just have questions or concerns, please let me know!

Independent Reviews

Both of these documents represent my own research and findings, with a few contributions from peers and associates. Other groups are doing their own research. I've shared earlier drafts of these lists with some of those other groups; most seem to use these lists as starting points for their own research. Having said that, I hope that these documents will help raise awareness of the risks associated with adopting new technologies without proper vetting.

I can't help but hum "I'm Got A Little List" from The Mikado whenever I work on these bullet points.

It should come as no surprise that I type a lot. And I really mean a lot. Almost every keyboard I own has worn off the letters on the keys. Then again, I type so much that I'm really a touch-typist; I rarely look at the keyboard. (Even before I took a class on typing on a typewriter in elementary school, I was already typing as fast I as I could talk. Do schools still teach typing?)

There are lots of different types of keyboards out there: high profile, low profile, feather sensitivity or heavy hitters, curved keys, uniform height, etc. Personally, I like the loud, heavy keyboards with high profiles (keys stand up) and no gap between the keys. This way, I can feel the keys without looking down and tell the difference between brushing a finger over a key and actually typing a letter. (If you can't hear my typing during a video call, then the keyboard isn't loud enough.)

Most of my keyboards have had black or gray keys. A few years ago (2022), I bought a "large print keyboard" for the fun of it. It had big keys, a high profile, and a loud click. The selling point for me were the huge letters and the bright yellow color.



Unfortunately, it didn't last very long. Within a few months, the black paint on the letters began to vanish. 'A', 'S', and 'D' were the first to go, followed by 'X', 'C', and 'V' (for cut, copy, and paste). Fast forward to today (3 years later):



The shift-key on the right doesn't have a black scratch on it. That's where I've literally worn through the yellow plastic. It's not that I don't use the letters Q, P, H, or U; they just seem to have lasted longer. (I joked with my colleagues that the backspace and delete keys are in pristine conditions -- because I don't make mistakes.)

The New Problems

When a keyboard gets worn down this much, I typically go out and buy a new cheap keyboard. Given that I wear through keyboards every few years, I have trouble justifying $100 for a fancy replacement. Give me a $10 cheap-plastic keyboard every few years and I'll be happy. (Seriously, I splurged $23 on the yellow keyboard. It lasted 3 years, so that's less than $8 a year. Before the yellow keyboard, I had a cheap $12 one that also lasted 3 years, so it cost $4 per year to use.)

Over the last 40+ years, I've seen the quality degrade as vendors cut costs by using cheaper materials. The old heavy IBM PC keyboards were built like tanks -- they never broke down, even if the letters might fade a little. The PS/2 keyboards (circa 1987-1997) had more plastic and occasionally the key switches would degrade before the print on the keys wore off. (I have one old PS/2 keyboard that types "jn" every time you press "n". Beneath each key is a switch. This problem might be a dirty contact, but I don't think I can open the keyboard up without breaking it.) Today's USB keyboards are extremely lightweight but also cheaply constructed; letters fade fast and the plastic on the keys might wear out. Today's keyboards are not built to last.

Making matters worse, most keyboards are made overseas. Between the (insane) tariffs and shipping delays, I don't want to wait. And with the current economic instability, I'd rather not spend the money, even on a new cheap keyboard, if I absolutely don't have to.

What's Old is New

Fortunately, I have a huge box of old keyboards in the storage area. It includes everything from modern USB to old PS/2 and the super old 5-pin DIN connectors. (I think the oldest keyboard in the box is from the early 1980s.) Some computer manufactures would bundle a keyboard with every new computer. Other times I'd pick up a keyboard in a box of auction junk. (Often, I'd want something else at the auction, but the box being sold also contained keyboards.) Any keyboard I don't like, don't need, don't use, or is broken for some reason gets put in the big box of keyboards.

Today I went digging through the box, looking for something with the right profile and feel.

• The first good one was a 105 keys with a PS/2 connector. (Most US keyboards have 101 keys.) My computer doesn't have a PS/2 port, but in the "big box of old keyboards" was an old PS2-to-USB adapter! That's the nice thing about keyboards -- they all use the same communication protocol. As long as you have the right adapter to plug it in, the computer will recognize it and it will just work.
  
  This new old keyboard was manufactured in 1992 by a company that no longer exists. (I looked them up. Today, there's a company with the same name, but they were founded in 2001.) And yet, the keyboard still works fine. Well, sort of. All of the standard "101" keys still work fine, but the custom "power", "sleep", "wake", and "Fn" buttons don't register when I press them. (Maybe I need to tweak the keyboard mapping? Probably not worth the effort.) Since it's not perfect, I went back to the box of keyboards.


• The next keyboard had a bunch of sticky keys that push down but pop up slowly. (From an auction, someone probably spilled a drink on the keyboard a few decades ago.)


• The original "Sun" keyboard looks like a PS/2 but doesn't work; it's probably not really communicating with PS/2. (When possible, stay away from proprietary connectors.)


• I found one of my old keyboards that I used with my OS/2 workstation. After plugging it in, I remembered why I replaced it: the space bar was broken. Many space bars have a metal wire that ensures that the key goes down evenly. The wire fits into some plastic clips underneath. After years of use, those clips had broken off.

I finally settled on an old HP keyboard that was buried in the box. It's a C1405B #ABA, manufactured in 1992, back when HP actually made keyboards. (OMG, sellers on Etsy and eBay call it "Vintage"!) It's a heavy monster and yellowed from age, but no wear on the letters. It has a good feel and every key seems to work.

There's just one problem. It predates the appearance of the "Super" key ("Windows" or "Command" key on keyboards, next to the shift buttons). On my desk are two computers that share the same keyboard and mouse: a Linux box and a Mac. I use some software called 'Synergy' to link the desktops. As the mouse goes off the side of one monitor, it appears on the next computer's screen. Linux doesn't use the Windows/Command key, but Macs do. This missing key is going to be a problem... Fortunately, Synergy permits me to remap 'alt' to the Mac 'Command' key. (Problem solved.)

Macros

On my desktop computer, I have a few macros mapped to certain keys. For example:

• I almost never use the function keys. I've mapped "F9" to toggle my mouse size. If I press it, then the cursor becomes larger -- which is great for video chats and sharing my screen -- the big icons help people to see my mouse. If I press F9 again, then the mouse returns to the normal small size.


• I've remapped the "Pause/Break" button. (In 40+ years, I've never used that button to pause/break anything.) Instead, it turns on/off the audio recorder on my telephone. With the push of a button, I can record any call to an MP3. (I use it to record spam phone calls; I wrote about the script back in 2014.) If the phone rings from an unknown caller, I press the button to record and then answer the phone. (And yes, recording calls is legal in Colorado.)


• The lower-right corner of most 101-key keyboards has a "Menu" button. I've remapped that to mute/unmute my speakers. (Sometimes I can't immediately find the app that is making sounds and I just want the computer to shut up while I take a call. Tap one key for mute.) However, this HP keyboard predates the "Windows" and "Menu" buttons, so I'll need to remap the mute/unmute to a different key. (Maybe F8; I never use that key!)

Unsurprisingly, the macros work with this new old keyboard. While the manufacturing quality has evolved over time, the keyboard communication protocol and key codes hasn't changed.

Old Learning Curve

I think the biggest hurdle for this new old keyboard will be my own adjusting to the physical key spacing. Cheap keyboards and older keyboards often use different key sizes. With this keyboard, the spacing is a little wider than the yellow keyboard. It also has a different sensitivity. (Not bad, just different.) Then again, if I decide I don't like it, then I can always go back to digging through my big box of old keyboards.

Today, FotoForensics turns 11 years old! When I first introduced FotoForensics, I didn't know if it would be used by anyone or even if the implementation would have problems with the load. (As I originally wrote, "I wonder how it will scale?") Today, it has received over 6,050,000 unique pictures (with over 800,000 in the last year) and it's in the top 80,000 of internet destinations (the exact position changes every few minutes, but it's around 80,000 right now). As far as scaling is concerned, it seems to be holding up well.

Science!

Even though the site is popular, there are always some people who wonder if it is "scientific" or if it really works. A quick search on Google Scholar turns up lots of scientific journal articles that discuss FotoForensics and Error Level Analysis. They all conclude that it does, in fact, work as advertised. Google Scholar returns over 400 results. Here is a random selection of examples:

• K., P. B. M., Singh, K., Pandey, S. S., & O'Kennedy, R. (2019). Identification of the forged images using image forensic tools. In Communication and computing systems: Proceedings of the 2nd International Conference on Communication and Computing Systems (ICCCS 2018), December 1-2, 2018, Gurgaon, India. essay, CRC Press.
  

  Abstract
  The contents of the digital images can be easily manipulated with image editing software like Adobe Photoshop, Pixelmator, Inkscape, Fireworks, etc. In real life applications, it is indispensable to check the authenticity of the digital images because forged images could deliver misleading information and messages to our community. Different tools have been developed to detect the forged images. In literature, there is no study which presents an insight into image forensic tools and their evaluation on the basis of different criteria. Therefore, to address this issue, we present an insight into digital image forensic tools; and evaluate it on the basis of 15 different parameters like “error level analysis”, “metadata analysis”, “JPEG luminance and chrominance data”, etc. For our experimental work, we choose “FotoForensics” tool to show the forged region in digital images; and JPEGsnoop tool has been used to extract the metadata of the images.



• Kageyama, K., Kumaki, T., Ogura, T., & Fujino, T. (2015). Digital image forensics using morphological pattern spectrum. Journal of Signal Processing, 19(4), 159–162. https://www.jstage.jst.go.jp/article/jsp/19/4/19_159/_article/-char/ja/


• Scheidt, N., Adda, M., Chateau, L., & Kutlu, Y. E. (2021). Forensic tools for IOT device investigations in regards to human trafficking. 2021 IEEE International Conference on Smart Internet of Things (SmartIoT). https://doi.org/10.1109/smartiot52359.2021.00010


• Almalki, S., Almalki, H., & Almansour, A. (2018, November). Detecting Deceptive Images in Online Content. In 2018 14th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS) (pp. 380-386). IEEE. https://ieeexplore.ieee.org/abstract/document/8706216

This is nowhere near the complete list. I'm seeing dozens of journal articles every year. Some evaluate FotoForensics, some use it to support conclusions, and others treat it as a baseline for evaluating new techniques. Moreover, those are just the articles that talk about "FotoForensics". The number of journal articles is even higher if I search for "Error Level Analysis".

Legal Use

"Forensics" means "for use in a court of law." When it comes to understanding forensic tools, the courts use a few criteria to determine if the tool or expert witness is qualified. In the United States, the criteria varies by state, but it's usually either the Daubert standard (from Daubert v. Merrell Dow Pharmaceuticals Inc., 509 U.S. 579 (1993)) or the Frye standard (from Frye v. United States, 293 F. 1013 (D.C. Cir. 1923)). In either case, there are five criteria for determining if evidence and expert testimony should be considered or accepted by the court. I think FotoForensics addresses each of them to the extreme:

1. Has the theory or technique in question been tested?
   In the case of FotoForensics, every algorithm and technique has been tested. Both by myself and by other experts in the field. The public FotoForensics service has a commercial counterpart. Every single one of the commercial customers has relied on their own independent tests before regularly using the tools.


2. Has it has been subjected to peer review and publication?
   This is a definite yes. It has both formally and informally been repeatedly subjected to peer review. While the original algorithms were published in a conference white paper, subsequent publications include this blog, the training material at FotoForensics, and the more than 400 third-party book and journal publications. (It's not just me writing about it.)


3. Does it have known or potential error rates?
   The question of "error rate" has always been difficult to answer. Confidence intervals are part of a statistical hypothesis. The cryptographic hashes from the Digest analyzer are good examples here. We can compute the SHA1 hash of two pictures and determine the likelihood of a mismatch. With cryptographic hashes, different hash values means that there were different input data sets. The likelihood of a false-negative match, where two byte-per-byte identical files are marked as being different, is zero (0); it doesn't happen. However, two different files could generate the same SHA1 hash value. The computed odds are about 1 in 2¹⁶⁰ (a huge number). It drops to 2⁸⁰ if we incorporate the Birthday Paradox.
   
   (Not all cryptographic hashes are the same. MD5 is considered 'weak'. A collision can be forced in around 2¹⁸ tries, or about 1 in 262,144.)
   
   In contrast, ELA, Hidden Pixels, Metadata, and the other data extraction systems do not use not a statistical hypothesis. These tools work like a microscope. What are the false-positive and false-negative rates for a microscope? It's a trick question; a microscope does not have them. As with other non-statistical systems, a microscope only identifies artifacts. The tests are deterministic and repeatable. It is up to a human to identify possible scenarios that are consistent with the observations. The documentation at FotoForensics identifies the various caveats and issues, but the tools never draw a conclusion. It's up to the human expert to evaluate the results and draw a conclusion.
   
   Since the various caveats and corner-case conditions are identified, it meets this requirement.


4. Are there existing and maintained standards controlling its operation?
   Yes. Most of the algorithms are documented and fixed (have not changed in a decade). If there is an implementation error, then we perform updates (maintenance). And some of the dependent applications, like ExifTool for metadata extraction, are regularly updated for detecting more information. This meets the criteria.


5. Has it attracted widespread acceptance within a relevant scientific community?
   Absolutely yes. Both the public and commercial versions are regularly used across a wide range of communities: mass media, financial, legal, insurance, UFO photo analysis (don't laugh), sales verification (receipt fraud is a big problem), contest validation, and more.

The courts also like to see historical precedence. The tools used at FotoForensics have been repeatedly used in legal cases. Everything from child custody battles and human trafficking to insurance and banking fraud. (I'm not the only expert using these tools.)

One Oversight

In my original (February 2012) announcement, I voiced some concerns about making tools publicly available. I was primarily concerned about possible misuse and the risks from educating criminals.

As for the tool being misused: I addressed this by releasing tutorials and challenges. Based on my web logs, these are some of the most popular documents I've ever written. Shortly after FotoForensics went live, a few trollish people posted bogus analysis on a wide range of topics to social media sites (Reddit, Twitter, Facebook, etc.). Each claimed that FotoForensics supported their arguments. I knew I did something right when those bogus claims would immediately be corrected by people who saw and understood the tutorials. (I don't have to police the internet; the community is doing that all by themselves.)

With regards to criminal behavior, I went so far as to write:

From an ethical viewpoint, I don't think this site violates concerns about educating criminals since (1) I don't distribute code, (2) bad guys generally don't like to submit their content to remote servers for evaluation, and (3) with the tutorial, people have the option to learn how to use the tool and are not left with a push-button solution.

Boy, was I wrong. Bad guys do like to submit their content to remote systems for evaluation! The public FotoForensics service regularly sees people developing new fraud techniques. Because new techniques stand out, I can often identify their tools and methods before they have a chance to deploy (weaponize) it for widespread fraud. Often, I can develop automated detectors before they distribute their forgery software. Over the years, I've written about everything from fraudulent scientific publications and government-sponsored techniques to widespread passport forgeries and commercially-sponsored fraud from Bayer and AvtoVaz.

FotoForensics is hardly a deploy-once-and-done service. I'm constantly learning new things and regularly improving it. I'm very thankful to my friends, partners, various collaborators, and the public for over a decade of helpful feedback, assistance, and insights. This year, I especially want to thank my mental support group (including Bill, Bob, and Dave), my totally technical support group (Marc, Jim, Richard, Wendy, Troy, and everyone else), Joe, Joe, Joe, AXT, the Masters and their wandering slaves, Evil Neal, Loris, and The Boss. Their advice, support, assistance, and feedback has been invaluable. And most importantly, I want to thank the literally millions of people who have used FotoForensics and helped make it what it is today.