Interview transcript:

 

Terry Gerton We’re sitting here after weeks of uncertainty and missed paychecks during the government shutdown and a lot of people are probably feeling kind of anxious about their finances. How does that stress from just day-to-day situations spill over into how people make decisions about investments?

Art Stein Well, stress and emotion make a big difference in how people make their investments. And with the TSP, it makes a big difference in how much people are putting in the stock funds, which are the C and the S and the I funds, and then how much they’re putting in, well, especially the G fund, which is a short-term bond fund, really, it’s more of a cash account. And you know, what I’ve seen time and again for 30 years is that when the stock market crashes, federal employees and retirees tend to get disgusted and move money into the G fund. And the problem with that is, there’s never a good time to take it out of the G fund and reinvest. Usually they’ve made that move after the market has declined and frequently don’t get back in until it’s gone back a lot. So really what we caution our clients to do is to set an investment plan. And part of the investment plan is to know what you’re going to do when the stock market does crash. Because inevitably it’s going to. We don’t know when. Stock market crashes average about one every four years or one every seven years, depending upon the time period, or somewhere in between. But they are a regular part of the market cycle. And what we mean by a stock market crash is that a particular stock market like the S&P 500, which is the basis for the C fund, goes down 20% or more from a previous high. And that’s also called a bear market. A bull market is when, let’s say, the S&P 500 increases more than 20% from a previous high. And people really avoid investing in stocks or putting too much money in stocks because they fear the bear markets, they fear the crashes, they don’t like the volatility. But we’re always having volatility in any market except a bank account or the G fund. Volatility is just a fluctuation in value. Now stocks are more volatile than bonds, that’s clear. But what investors should do is trying to determine appropriate allocation between stock investments and bond investments and bank accounts. And the TSP, that means what percentage of your investments do you want in the G and the F funds, which are bonds and cash accounts, and what percentage do you want in stocks, which are C, S and I? And once you choose that percent, stick with it unless there’s a good reason to change. And the stock market crash is not really a good reason to change. And if the stock market crashes, especially for employees, that’s an opportunity. They’re investing money every two weeks. And of course they’d rather buy shares in the C and the S and the I funds when those are down and cheap than when they’re high and expensive. So just being able to stick to it really makes a difference.

Terry Gerton It’s really hard to imagine that the market is going to crash anytime soon. It’s been on such a steady upward climb for so many months. And yet you talk about when that correction, which is impossible to predict exactly, but pretty possible to predict generally happens, people do the opposite of standard recommendation. They sell low and then try to buy again high instead of buying low and selling high. Talk to us again about what kind of planning can help people avoid the emotional response to that sort of occurrence.

Art Stein Well, I think it’s very important to one, know and admit to yourself and take into account that the market’s going to crash. I mean, it’s going to happen. And it’s not unusual. It’s typical. And two, especially for employees, don’t change your investment allocation if the stock markets crash, unless you’re increasing your percentage allocation of your biweekly investments into the TSP fund. If you’re increasing the percentage going into the stock funds, that would make sense. And, you know Terry, when we speak to TSP millionaires, one consistent theme is that they had most of their investments going to the stock funds. And they did not change that when the stock markets crashed. They just kept investing. They accepted that. It was a long-term investment. And they just stuck with it.

Terry Gerton I’m speaking with certified financial planner Art Stein of Arthur Stein Financial. Art, we’re talking about a disciplined, non-emotional approach to investment here, but we’ve just come out of the longest government shutdown in history. And the current continuing resolution only goes through the 30th of January, about two and a half months from now. So how should feds think not just about their investments, about building up or building back their emergency savings if they had to dip into it during the shutdown?

Art Stein Well, this shutdown was horrible, as we know. People were living on credit card debt in many cases. It shows how important it is to have an emergency fund, three to six months of expenses in a bank account, or maybe the G fund. And what we sometimes have to recommend to people, we don’t like doing it, is to reduce your contributions to the TSP to 5%. Because in many cases, Terry, we’re speaking to people who are maxing out their contributions. But no, if you don’t have an emergency fund, that’s a mistake. Reduce it to 5%. Don’t go below that because you want to get the full 5% match from the federal government. Take that extra money that you were investing and use it to build up a bank account, three to six months of expenses. And especially, you know, this is so crazy. We’ve gone through this long shutdown, and then they had this big victory. But when you look at the victory, it only funded the government for two and a half months. I mean, how short term is that? So now is a good time. Just get on the TSP website and reduce your contributions to 5% and build up some cash. I mean, I’m praying and hoping that they won’t do another shutdown on you know, January 30th, but as we all know, things are not good with these negotiations.

The post An objective, unemotional investment strategy for your TSP, easy to say but hard to do in uncertain times first appeared on Federal News Network.

© AP Photo/David Dermer

Democratic lawmakers from the US House of Representatives have accused President Donald Trump and his administration of using the White House to enrich the presidential family through their crypto businesses, calling on Congress to fight corruption.

House Democrats Call Out ‘New Age of Corruption’

In a recent report from the House Judiciary Committee, Democratic Representative Jamie Raskin claimed that President Trump had allegedly “exploited” the presidency and transformed the White House into “a personal money-making operation” that has added billions of dollars to his net worth through his crypto ventures.

The 27-page document, named “Trump, Crypto, and a New Age of Corruption,” compiled multiple news media outlet reports affirming that the US President has seen his family’s crypto holdings surge to $11.6 billion since he stepped into office in January, making over $800 million from the sale of digital assets in the first half of 2025.

As reported by Bitcoinist, a Financial Times investigation released in October claimed that the Trump family’s crypto fortune has surged to over $1 billion from his multiple digital asset ventures since his return to the White House.

The investigation explained that Trump’s digital asset businesses have significantly boosted the US President’s net worth on paper by billions of dollars, but only calculated the potential income from the realized profits of World Liberty Financial’s WLFI token and USD1 stablecoin, and the official TRUMP and MELANIA memecoins.

According to FT calculations, the TRUMP and MELANIA memecoins made around $362 million and $65 million, respectively, for a total of $427 million in sales and trading fees. Moreover, the WLFI token had generated approximately $550 million by the time of the investigation, while the USD1 stablecoin, which recorded $2.71 billion in total sales, potentially $42 million.

“Donald Trump has turned the Oval Office into the world’s most corrupt crypto startup operation, minting staggering personal fortunes for him and his family in less than a year.” Representative Raskin affirmed in a Press Release.

“We don’t know where all the money is coming from yet, but America has never seen corruption on this scale take place inside the White House. This Report shows how Trump’s so-called ‘pro-crypto agenda’ is just one more Trump family self-enrichment plan, built on pay-to-play deals and corrupt foreign interests seeking secret channels of access and influence,” he continued.

Trump’s Crypto Ventures Expose ‘Weaknesses’ Of US System

Raskin also stated that the Trump administration has “dismantled” federal oversight and safeguards that were set to protect Americans from fraud, scams, and financial exploitation. “Trump has been pardoning criminals who commit fraud through crypto and dismantling the regulations that protect legitimate American investors.”

The report presented a list of actions that advanced “Trump and his family’s personal financial interests at the expense of the law, ethics, and national security.” The evidence exposed in the document included claims that the President’s crypto ventures have “attracted substantial investments from foreign nationals and state-linked entities seeking to curry favor with the Administration.”

Last week, Democratic Senators expressed concerns about potential national security risks related to World Liberty Financial over token sales allegedly linked to illicit actors. In a letter, the lawmakers requested that Attorney General Pam Bondi and Treasury Secretary Scott Bessent investigate allegations that the Trump-backed company had sold tokens to sanctioned entities or individuals with ties to illicit actors in Russia and North Korea.

They also argued that World Liberty Financial and its token “lacks adequate safeguards to prevent bad actors from moving funds or gaining influence over its governance,” which raises concerns over a potential conflict of interest.

The Committee’s report highlighted that the Trump Administration has paused or ended most investigations and enforcement actions involving major crypto companies and disbanded multiple federal-level crypto enforcement units. In addition, it expressed concern about the controversial pardons issue to key industry players, like Binance’s co-founder, Changpeng Zhao.

Ultimately, the document warned that Trump’s crypto moves seemingly expose “severe weaknesses” in the US’s campaign finance, conflict-of-interest, and anti-bribery laws. “Congress must expose this dangerous grift, and defend the rule of law against the profiteers and criminals who would destroy it,” Raskin concluded.

Is it located in the same place as the plan to repeal and replace Obamacare?

Last month, one of my PTSD (People Tech Support Duties) requests led me down a deep path related to AI-alterations inside images. It began with a plea to help photograph a minor skin irritation. But this merged with another request concerning automated AI alterations, provenance, and detection. Honestly, it looks like this rush to embrace "AI in everything" is resulting in some really bad manufacturer decisions.

What started with a request to photograph a minor skin rash ended up spiraling into a month-long investigation into how AI quietly rewrites what we see.

Cameras Causing Minor Irritations

The initial query came from a friend. Her kid had a recurring rash on one arm. These days, doctor visits are cheaper and significantly faster when done online or even asynchronously over email. In this case, the doctor sent a private message over the hospital's online system. He wanted a photo of the rash.

This sounds simple enough. Take out the camera, hold out the arm, take a photo, and then upload it to the doctor. Right?

Here's the problem: Her camera kept automatically applying filters to make the picture look better. Visually, the arm clearly had a rash. But through the camera, the picture just showed regular skin. It was like one of those haunted house scenes where the mirror shows something different. The camera wasn't capturing reality.



These days, smart cameras often automatically soften wrinkles and remove skin blemishes -- because who wants a picture of a smiling face with wrinkles and acne? But in this case, she really did want a photo showing the skin blemishes. No matter what she did, her camera wouldn't capture the rash. Keep in mind, to a human seeing it in real life, it was obvious: a red and pink spotted rash over light skin tone. We tried a couple of things:

• Turn off all filters. (There are some hidden menus on both Android and iOS devices that can enable filters.) On the Android, we selected the "Original" filter option. (Some Androids call this "None".) Nope, it was still smoothing the skin and automatically removing the rash.


• Try different orientations. On some devices (both Android and iOS), landscape and portrait modes apply different filters. Nope, the problem was still present.


• Try different lighting. While bright daylight bulbs (4500K) helped a little, the camera was still mitigating most of it.


• Try a different camera. My friend had both an Android phone and an Apple tablet; neither was more than 3 years old. Both were doing similar filterings.

We finally did find a few ways to get good pictures of the rash:

• Use a really old digital camera. We had a 10+ year old Sony camera (not a phone; a real standalone camera). With new batteries, we could photograph the rash.


• On my older iPhone 12 mini, I was able to increase the exposure to force the rash's red tint to stand out. I also needed bright lighting to make this work. While the colors were far from natural, they did allow the doctor to see the rash's pattern and color differential.


• My laptop has a built-in camera that has almost no intelligence. (After peeling off the tape that I used to cover the camera...) We tried a picture and it worked well. Almost any desktop computer's standalone webcam, where all enhancements are expected to be performed by the application, should be able to take an unaltered image.

I'm glad my friend's kid found this entire experimentation process fascinating. But if this had been a more time-sensitive issue, I honestly don't know what a typical user with a newer device could have done.

This irritating experience was just a scratch of a much larger issue that kept recurring over the month. Specifically, how modern cameras' AI processing is quietly rewriting reality.

AI Photos

Since the start of digital photography, nearly all cameras have included some form of algorithmic automation. Normally it is something minor, like auto-focus or auto-contrast. We usually don't think of these as being "AI", but they are definitely a type of AI. However, it wasn't until 2021 when the first camera-enabled devices with smart-erase became available. (The Google Pixel 6, Samsung Galaxy S21, and a few others. Apple didn't introduce its "Clean Up" smart erase feature until 2024.)

Following the rash problem, I had multiple customer requests asking whether their pictures were real or AI. Each case concerned the same camera: The new Google Pixel 10. This is the problem that I predicted at the beginning of last month. Specifically, every picture from the new Google Pixel 10 is tagged by Google as being processed by AI. This is not something that can be turned off. Even if you do nothing more than bring up the camera app and take a photo, the picture is tagged with the label:

Digital Source Type: http://cv.iptc.org/newscodes/digitalsourcetype/computationalCapture

According to IPTC, this means:

The media is the result of capturing multiple frames from a real-life source using a digital camera or digital recording device, then automatically merging them into a single frame using digital signal processing techniques and/or non-generative AI. Includes High Dynamic Range (HDR) processing common in smartphone camera apps.

In other words, this is a composite image. And while it may not be created using a generative AI system ("and/or"), it was definitely combined using some kind of AI-based system.

In industries that are sensitive to fraud, including banking, insurance, know-your-customer (KYC), fact checking, legal evidence, and photojournalism, seeing any kind of media that is explicitly labeled as using AI is an immediate red flag. What's worse is that analysis tools that are designed to detect AI alterations, including my tools and products from other developers, are flagging Pixel 10 photos as being AI. Keep in mind: Google isn't lying -- every image is modified using AI and is properly labeled. The problem is that you can't turn it off.

One picture (that I'm not allowed to share) was part of an insurance claim. If taken at face value, it looked like the person's car had gone from 60-to-zero in 0.5 seconds (but the tree only sustained minor injuries). However, the backstory was suspicious and the photos, from a Google Pixel 10, had inconsistencies. Adding to these problems, the pictures were being flagged as being partially or entirely AI-generated.

We can see this same problem with a sample "original" Pixel 10 image that I previously used.



At FotoForensics, the Error Level Analysis (ELA) permits visualizing compression artifacts. All edges should look similar to other edges, surfaces should look like surfaces, and similar textures should look similar. With this image, we can see a horizontal split in the background, where the upper third of the picture is mostly black, while the lower two thirds shows a dark bluish tinge. The blue is due to a chrominance separation, which is usually associated with alterations. Visually, the background looks the same above and below (it's the same colors above and below), so there should not be a compression difference. The unexpected compression difference denotes an alteration.



The public FotoForensics service has limited analyzers. The commercial version also detects:

• A halo around the light fixture, indicating that either the background was softened or the chandelier was added or altered. (Or all of the above.)


• The chevrons in the stained glass were digitally altered. (The Pixel 10 boosted the colors.)


• The chandelier has very strong artifacts that are associated with content from deep-learning AI systems.

None of these were intentional alterations. (Jeff just opened the camera app and took a picture. Nothing fancy by the human.) These are all AI-alterations by the Google Pixel 10 and they cannot be disabled.

In my previous blog entry, I showed that Google labels all photos as AI and that the metadata can be altered without detection. But with these automatic alterations baked into the image, we can no longer distinguish reality from revision.

Were the pictures real? With the car photos (that I cannot include here), my professional opinion was that, ignoring the AI and visual content, the photos were being misrepresented. (But doesn't the Pixel 10 use C2PA and sign every photo? Yes it does, but it doesn't help here because the C2PA signatures don't protect the metadata.) If I ignored the metadata, I'd see the alterations and AI fingerprints, and I'd be hard-pressed to determine if the detected artifacts were human initiated (intentional) or automated (unintentional). This isn't the desired AI promise, where AI generates content that looks like it came from a human. This is the opposite: AI forcing content from a human to look like AI.

Other Tools

After examining how these AI-enabled systems alter photos, the next question becomes: how well can our current tools even recognize these changes?

My analysis tools rely on deterministic algorithms. (That's why I call the service "FotoForensics" -- "Forensics" as in, evidence suitable for a court of law.) However, there are other online services that use AI to detect AI. Keep in mind, we don't know how well these AI systems were trained, what they actually learned, what biases they have, etc. This evaluation is not a recommendation to use any of these tools.

This inconsistency between different AI-based detection tools is one of the big reasons I don't view any of them as serious analyzers. For the Pixel 10 images, my clients had tried some of these systems and saw conflicting results. For example, using the same "original" Pixel 10 baseline image:

• Hive Moderation trained their system to detect a wide range of specific AI systems. They claim a 0% chance that this Pixel 10 photo contains AI, because it doesn't look like any of the systems they had trained on. Since the Pixel 10 uses a different AI system, they didn't detect it.
  
  


• Undetectable AI gives no information about what they detect. They claim this picture is "99% REAL". (Does that mean it's 1% fake?)
  
  


• SightEngine decided that it was "3%" AI, with a little generative AI detected.
  
  


• Illuminarty determined that it was "14.9%" AI-generated. I don't know if that refers to 14.9% of the image, or if that is the overall confidence level.
  
  


• At the other extreme, Was It AI determined that this Google Pixel 10 picture was definitely AI. It concluded: "We are quite confident that this image, or significant part of it, was created by AI."
  
  

The ground truth is that the Pixel 10 always uses AI to auto-enhance the picture. If you work in a field that forbids any AI enhancement, then the Pixel 10 is a serious problem. (You can't just tell your client that they need to go back to the site of the accident and take pictures with a different camera.)

Fear the Future

Once upon a time, "taking a picture" meant pressing a button and capturing something that looked like reality. Today, it's more like negotiating with an algorithm about what version of reality it's willing to show you. The irony is that the more "intelligent" cameras become, the less their output can be trusted. When even a simple snapshot passes through layers of algorithmic enhancement, metadata rewriting, and AI tagging, the concept of an "original" photo starts to vanish.

People use AI for lots of tasks these days. This includes helping with research, editing text, or even assisting with diagnostics. However, each of these uses still leaves the human with the final decision about what to accept, reject, or cross-validate. In contrast, the human photographer has no option to reject the AI's alterations to these digital photos.

From medical photos and insurance claims to legal evidence, the line between "photo" and "AI-enhanced composite" has blurred. For fields that rely on authenticity, that's not a minor inconvenience; it's a systemic problem. Until manufacturers return real control to the photographer, sometimes the most reliable camera is the old one in the junk drawer -- like a decade-old Sony camera with no Wi-Fi, no filters, and no agenda.

P.S. Brain Dead Frogs turned this blog entry in a song for an upcoming album. Enjoy!

A few months ago I wrote about my experimentation this year with solar power. I thought I would give a couple of updates.

The basic architecture hasn't changed, but some of the components have:



Given that I've never done this before, I expected to have some problems. However, I didn't expect every problem to be related to the power inverter. The inverter converts the 12V DC battery's power to 120V AC for the servers to use. Due to technical issues (none of which were my fault), I'm currently on my fourth power inverter.

Inverter Problem #1: "I'm Bond, N-G Bond"

The first inverter that I purchased was a Renogy 2000W Pure Sine Wave Inverter.



This inverter worked fine when I was only using the battery. However, if I plugged it into the automated transfer switch (ATS), it immediately tripped the wall outlet's circuit breaker. The problem was an undocumented grounding loop. Specifically, the three-prong outlets used in the United States are "hot", "neutral", and "ground". For safety, the neutral and ground should be tied together at one location; it's called a neutral-ground bond, or N-G bond. (For building wiring, the N-G bond is in your home or office breaker box.) Every outlet should only have one N-G bond. If you have two N-G bonds, then you have a grounding loop and an electrocution hazard. (A circuit breaker should detect this and trip immediately.)

The opposite of a N-G bond is a "floating neutral". Only use a floating neutral if some other part of the circuit has the N-G bond. In my case, the automated transfer switch (AFS) connects to the inverter and the utility/wall outlet. The wall outlet connects to the breaker box where the N-G bond is located.

What wasn't mentioned anywhere on the Amazon product page or Renogy web site is that this inverter has a built-in N-G bond. It will work great if you only use it with a battery, but it cannot be used with an ATS or utility/shore power.

There are some YouTube videos that show people opening the inverter, disabling the N-G bond, and disabling the "unsafe alarm". I'm not linking to any of those videos because overriding a safety mechanism for high voltage is incredibly stoopid.

Instead, I spoke to Renogy's customer support. They recommended a different inverter that has an N-G bond switch: you can choose to safely enable or disable the N-G bond. I contacted Amazon since it was just past the 30-day return period. Amazon allowed the return with the condition that I also ordered the correct one. No problem.

The big lesson here: Before buying an inverter, ask if it has a N-G bond, a floating neutral, or a way to toggle between them. Most inverters don't make this detail easy to find. (If you can't find it, then don't buy the inverter.) Make sure the configuration is correct for your environment.

• If you ever plan to connect the inverter to an ATS that switches between the inverter and wall/utility/shore power, then you need an inverter that supports a floating neutral.


• If you only plan to connect the inverter to a DC power source, like a battery or generator, then you need an inverter that has a built-in N-G bond.

Inverter Problem #2: It's Wrong Because It Hertz

The second inverter had a built-in switch to enable and disable the N-G bond. The good news it that, with the N-G bond disabled, it worked correctly through the ATS. To toggle the ATS, I put a Shelly Plug smart outlet between the utility/wall outlet and the ATS.



I built my own controller and it tracks the battery charge level. When the battery is charged enough, the controller tells the inverter to turn on and then remotely tells the Shelly Plug to turn off the wall outlet. That causes the ATS to switch over to the inverter.

Keep in mind, the inverter has it's own built-in transfer switch. However, the documentation doesn't mention that it is "utility/shore priority". That is, when the wall outlet has power, the inverter will use the utility power instead of the battery. It has no option to be plugged into a working outlet and to use the battery power instead of the outlet's power. So, I didn't use their built-in transfer switch.

This configuration worked great for about two weeks. That's when I heard a lot of beeping coming from the computer rack. The inverter was on and the wall outlet was off (good), but the Tripp Lite UPS feeding the equipment was screaming about a generic "bad power" problem. I manually toggled the inverter off and on. It came up again and the UPS was happy. (Very odd.)

I started to see this "bad power" issue about 25% of the time when the inverter turned on. I ended up installing the Renogy app to monitor the inverter over the built-in Bluetooth. That's when I saw the problem. The inverter has a frequency switch: 50Hz or 60Hz. The switch was in the 60Hz setting, but sometimes the inverter was starting up at 50Hz. This is bad, like, "fire hazard" bad, and I'm glad that the UPS detected and prevented the problem. Some of my screenshots from the app even showed it starting up low, like at 53-58 Hz, and then falling back to 50Hz a few seconds later.


(In this screenshot, the inverter started up at 53.9Hz. After about 15 seconds, it dropped down to 50Hz.)

I eventually added Bluetooth support to my homemade controller so that I could monitor and log the inverter's output voltage and frequency. The controller would start up the inverter and wait for the built-in Bluetooth to come online. Then it would read the status and make sure it was at 60Hz (+/- 0.5Hz) and 120V (+/- 6V) before turning off the utility and transferring the load to the inverter. If it came up at the wrong Hz, the controller would shut down the inverter for a minute before trying again.

It took some back-and-forth discussions with the Renogy technical support before they decided that it was a defect. They offered me a warranty-exchange. It took about two weeks for the inverter to be exchanged (one week there, one week back). The entire discussion and replacement took a month.

The replacement inverter was the same make and model. It worked great for the first two weeks, then developed the exact same problem! But rather than happening 25% of the time, it was happening about 10% of the time. To me, this looks like either a design flaw or a faulty component that impacts the entire product line. The folks at Renogy provided me with a warranty return and full refund.

If you read the Amazon reviews for the 2000W and 3000W models, they have a lot of 1-star reviews with comments about various defects. Other forums mention that items plugged into the inverter melted and motors burned out. Melting and burned out motors are problems that can happen if the inverter is running at 50Hz instead of 60Hz.

The Fourth Inverter

For the fourth inverter, I went with a completely different brand: a Landerpow 1500W inverter. Besides having what I needed, it also had a few unexpectedly nice benefits compared to the Renogy:

• I had wanted a 2000W inverter, but a 1500W inverter is good enough. Honestly, my servers are drawing about 1.5 - 2.5 amps, so this is still plenty of overkill for my needs. The inverter says it can also handle surges of up to 3000W, so it can easily handle a server booting (which draws much more power than post-boot usage).


• The documentation clearly specifies that the Landerpow does not have an N-G bond. That's perfect for my own needs.


• As for dimensions, it's easily half the size of the Renogy 2000W inverter. The Landerpow also weighs much less. (When the box first arrived, I thought it might be empty because it was so lightweight.)


• The Renogy has a built-in Bluetooth interface. In contrast, the Landerpow doesn't have built-in Bluetooth. That's not an issue for me. In fact, I consider Renogy's built-in Bluetooth to be a security risk since it didn't require a login and would connect to anyone running the app within 50 feet of the inverter.


• The Landerpow has a quiet beep when it turns on and off, nothing like Renogy's incredibly loud beep. (Renogy's inverter beep could be heard outside the machine room and across the building.) I view Landerpow's quiet beep as a positive feature.


• With a fully charged battery and with no solar charging, my math said that I should get about 5 hours of use out of the inverter:
  
  
  • The 12V, 100Ah LiFePO4 battery should provide 10Ah at 120V. (That's 10 hours of power if you're using 1 amp.)
  
  
  • There's a DC-to-AC conversion loss around 90%, so that's 9Ah under ideal circumstances.
  
  
  • You shouldn't use the battery below 20% or 12V. That leaves 7.2Ah usable.
  
  
  • I'm consuming power at a rate of about 1.3Ah at 120V. That optimistically leaves 5.5 hours of usable power.
  
  With the same test setup, none of the Renogy inverters gave me more than 3 hours. The Landerpow gave me over 5 hours. The same battery appears to last over 60% longer with the Landerpow. I don't know what the Renogy inverter is doing, but it's consuming much more battery power than the Landerpow.


• Overnight, when there is no charging, the battery equalizes, so the voltage may appear to change overnight. Additionally, the MPPT and the controller both run off the battery all night. (The controller is an embedded system requires 5VDC and the MPPT requires 9VDC; combined, it's less than 400mA.) On top of this, we have the inverter connected to the battery. The Landerpow doesn't appear to cause any additional drain when powered off. ("Off" means off.) In contrast, the Renogy inverter (all of them) caused the battery to drain by an additional 1Ah-2Ah overnight. Even though nothing on the Renogy inverter appears to be functioning, "off" doesn't appear to be off.


• The Renogy inverter required a huge surge when first starting up. My battery monitor would see it go from 100% to 80% during startup, and then settle at around 90%-95%. Part of this is the inverter charging the internal electronics, but part is testing the fans at the maximum rating. In contrast, the Landerpow has no noticeable startup surge. (If it starts when the battery is at 100% capacity and 13.5V, then it will still be at 100% capacity and 13.5V after startup.) Additionally, then Landerpow is really quiet; it doesn't run the fans when it first turns on.

The Renogy inverter cost over $300. The Landerpow is about $100. Smaller, lighter, quieter, works properly, consumes less power, and less expensive? This is just icing on the cake.

Enabling Automation

My controller determines when the inverter should turn on/off. With the Renogy, there's an RJ-11 plug for a wired remote switch. The plug has 4 wires (using telephone coloring, that's black, red, green, and yellow). The middle two wires (red and green) are a switch. If they are connected, then the inverter turns on; disconnected turns it off.

The Landerpow also has a four-wire RJ-11 connector for the remote. I couldn't find the pinout, but I reverse-engineered the switch in minutes.

The remote contains a display that shows voltage, frequency, load, etc. That information has to come over a protocol like one-wire, I2C (two wire), UART (one or two wire), or a three wire serial connection like RS232 or RS485. However, when the inverter is turned off, there are no electronics running. That means it cannot be a communication protocol to turn it on. I connected my multimeter to the controller and quickly found that the physical on/off switch was connected to the green-yellow wires. I wired that up to my controller's on/off relay and it worked perfectly on the first try.

I still haven't worked out the communication protocol. (I'll save that for another day, unless someone else can provide the answer.) At minimum, the wires need to provide ground, +5VDC power for the display, and a data line. I wouldn't be surprised if they were using a one-wire protocol, or using the switch wires for part of a serial communication like UART or RS485. (I suspect the four wires are part of a UART communication protocol: black=ground, red=+5VDC, green=data return, and yellow=TX/RX, with green/yellow also acting as a simple on/off switch for the inverter.)

Pictures!

I've mounted everything to a board for easy maintenance. Here's the previous configuration board with the Renogy inverter:



And here's the current configuration board with the Landerpow inverter:



You can see that the new inverter is significantly smaller. I've also added in a manual shutoff switch to the solar panels. (The shutoff is completely mounted to the board; it's the weird camera angle that makes it look like it's hanging off the side.) Any work on the battery requires turning off the power. The MPPT will try to run off solar-only, but the manual warns about running from solar-only without a battery attached. The shutoff allows me to turn off the solar panels before working on the battery.

Next on the to-do list:

• Add my own voltmeter so the controller can monitor the battery's power directly. Reading the voltage from the MPPT seem to be a little inaccurate.


• Reverse-engineering the communication to the inverter over the remote interface. Ideally, I want my own M5StampS3 controller to read the inverter's status directly from the inverter.

As components go, the Renogy solar panels seem very good. The Renogy MPPT is good, but maybe not the best option. Avoid Renogy inverters and consider the Landerpow inverter instead. I'm also a huge fan of Shelly Plugs for smart outlets and the M5StampS3 for the DIY controller.

Efficiency

Due to all of the inverter problems, I haven't had a solid month of use from the solar panels yet. We've also had a lot of overcast and rainy days. However, I have had some good weeks. A typical overcast day saves about 400Wh per day. (That translates to about 12kWh/month in the worst case.) I've only had one clear-sky day with the new inverter, and I logged 1.2kWh of power in that single day. (A month of sunny days would be over 30kWh in the best case.) Even with partial usage and overcast skies, my last two utility bills were around 20kWh lower than expected, matching my logs -- so this solar powered system is doing its job!

I've also noticed something that I probably should have realized earlier. My solar panels are installed as awnings on the side of the building. At the start of the summer, the solar panels received direct sunlight just after sunrise. The direct light ended abruptly at noon as the sun passes over the building and no longer hit the awnings. They generate less than 2A of power for the rest of the day through ambient sunlight.

However, we're nearing the end of summer and the sun's path through the sky has shifted. These days, the panels don't receive direct light until about 9am and it continues until nearly 2pm. By the time winter rolls around, it should receive direct light from mid-morning until a few hours before sunset. The panels should be generating more power during the winter due to their location on the building and the sun's trajectory across the sky. With the current "overcast with afternoon rain", I'm currently getting about 4.5 hours a day out of the battery+solar configuration. (The panels generate a maximum of 200W, and are currently averaging around 180W during direct sunlight with partially-cloudy skies.)

I originally allocated $1,000 for this project. With the less expensive inverter, I'm now hovering around $800 in expenses. The panels are saving me a few dollars per month. At this rate, they will probably never pay off this investment. However, it has been a great way to learn about solar power and DIY control systems. Even with the inverter frustrations, it's been a fun summer project.

I'm continuing to look for ways to lower my energy bill, even if only by a few dollars. One of my ideas was to use solar panels. However, the roof on the office building isn't ideal for solar.

• The optimal direction is East to South-East for morning and South-West to West for afternoon. Unfortunately, the southern facing parts of the roof have lots of small sections, so there's no place to mount a lot of solar panels. But I do have space for a few panels on the roof; probably enough to power the server rack.


• All of the professional solar installation companies either don't want to install panels if it's less than 100% of your energy needs, or they want to charge so much that it won't be worth the installation costs. This rules out the "few solar panels" option from a professional installer.

Last year, I decided that it would be a good learning experience to make my own solar panel Energy Storage System (ESS). My goal was not to power the entire office or sell power back to the electric company. Rather, I wanted an off-grid solution to just power the server rack for a few hours each day. If it worked, it should save me somewhere between 20kWh and 40kWh per month. That's less than 10% of my utility bill, but it's better than nothing. And assuming I ran the numbers correctly, it should pay itself off in about 5 years. (I hoped to keep the costs significantly lower by doing the installation by myself.)

In the worst case, it may never earn enough to pay itself off. But at least I'll learn something about solar panels, energy storage systems, and high voltage.

Having said that, developing it myself was certainly full of unexpected surprises and learning curves. Each time I thought I had everything I needed, I ended up finding another problem. (Now I know why professional installers charge tens of thousands of dollars. I don't even want to think about how much of my labor that went into this.)

The Basic Idea

I started this project with a basic concept. For the rest of the details, I decided that I'd figure it out as I went along.

1. Goal: I want an off-grid solar powered system for my server rack. It is not intended to run 24 hours a day, cover all of my energy needs, or sell excess power back to the utilities. I only want to reduce my power usage and related costs by a little. (When I consulted with professional solar installers, this is a concept that they could not comprehend.)


2. Low budget: A professional installation can cost over $20,000. I want to keep it under $1,000. For me, I wanted this to be a learning experience that included solar power and embedded controllers.


3. Roof: The original plan was to put some panels on the roof. Since I don't have much roof space, I was only going to have two panels that, under ideal conditions, could generate about 100 watts of power each, forming a 200W solar system. This won't power the entire office, but it should power the server rack for a few hours each day. (I ended up not going with a roof solution, but I'll cover that in a moment.)

The entire architecture is kind of overwhelming. Here's a drawing that shows what I'm doing:



And here's the final system:



Note: I'm naming a lot of brands to denote what I finally went with. This is not an endorsement or sponsorship; this is what (eventually) worked for me. I'm sure there are other alternatives, and I didn't necessarily choose the least expensive route. (This was a learning experience.)

• Solar charger: The solar panels connect to a battery charger, or Maximum Power Point Tracking (MPPT) system. The MPPT receives power from the solar cells and optimally charges the battery. Make sure to get an MPPT that can handle all of the power from your panels! My MPPT is a Renogy Rover 20, a 20-amp charger that can handle a wide range of batteries. The two black wires coming out the bottom go to the battery. There's also a thin black line that monitors the battery's temperature, preventing overcharging and heat-related problems. Coming off the left side are two additional black lines that connect to the solar panels. (The vendor only included black cables. I marked one with red electrical tape so I could track which one carried the positive charge.) There's also a 10-amp fuse (not pictured) from the solar panels to the MPPT.


• Battery: The MTTP receives power form the panels and charges up a moderately large battery: 12V 100Ah LiFePO4 deep cycle battery. (Not pictured; it's in the cabinet.) When fully charged, the battery should be able to keep the servers running for about 30 minutes.


• Inverter: On the right is a Renogy 2000W power inverter. It converts the 12V DC battery into 120V 60Hz AC power. It has two thick cables that go to the battery, with red going through a 20-amp fuse. (Always put fuses on the red/positive lines.)


• Automatic Transfer Switch (ATS): At the top (yellow box) is the automatic transfer switch (ATS) that toggles between utility/wall power and the inverter's power. It has a 30ms transfer speed. I had been using this box for years to manually switch between power sources without interruption. The three cables coming out of it go to the two inputs: primary is the inverter and fallback is the wall outlet. The output line goes to the UPS in the server rack. The UPS ensures that there isn't an outage during the power transfer. It also includes a power smoother to resolve any potential power spikes or phase issues.


• Output power: The ATS's output AC power (from grid or inverter) goes into a smart outlet (not pictured in the line drawing, but visible in the photo below as a white box plugged into the yellow connector at the top). This allows me to measure how much power the server rack consumes. There's a second smart outlet (not pictured) between the wall outlet and the ATS, allowing me to measure the power consumption from the utility. When I'm running off grid power, both smart outlets report the same power consumption (+/- a few milliamps). But when I'm running off the inverter, the grid usage drops to zero.


• Controller: In the middle (with the pretty lights) is my DIY embedded controller. It reads the battery level and charging state from the MPPT and has a line that can remotely turn on and off the inverter. It decides when the inverter runs based on the battery charge level and available voltage. It also has a web interface so I can query the current status, adjust parameters, and manually override when it runs.


• Ground: Not seen in the picture, there's grounding wire from the inverter's external ground screw to the server rack. The server rack is tied to the building's "earth ground". Proper grounding is essential for safety.

Everything is mounted vertically to a board that is hung from the side of the server rack. This allows me to easily take it down for any maintenance issues. (And when doing the initial testing, I could carry the entire thing outside.)

Even though I knew I'd be starting this project around March of this year, I started ordering supplies five months earlier (last November). This included solar panels, a solar charger, battery, and an inverter. I ordered other components as I realized I needed them. Why did I start this so early? I believed Trump when he said he would be imposing stiff tariffs, making everything more expensive. (In hindsight, this was a great decision. If I started ordering everything today, some items would cost nearly twice as much.)

Measuring Power

Before starting this project, I needed to understand how much power I'd require and how much it might save me on my utility bill.

As a software (not hardware) person, I'm definitely not an electrical engineer. For you non-electricians, there are three parts of electricity that need to be tracked:

• Voltage (V). This is the amount of power supplied on the wires. Think of it like the pressure in a water pipe.


• Amps (A). This is the amount of current available. Think of this like the size of the water pipe. A typical desktop computer may require a few amps of power. Your refrigerator probably uses around 20 amps when the compressor is running, while an IoT embedded device usually uses 200mA (milliamps, or 0.2A, that's flea power).


• Watts (W). This is the amount of work available. W=A×V.

These measurements are often compared to flowing water. Volts identify how fast a river is flowing (the water pressure). Amps identify how large the river is, and watts is the total energy delivered by the river. A wide but slow moving river has high amps but low voltage. A narrow but fast flowing river has a low current (low amps) but a high voltage. Because of the relationship between W, A, and V, the electronics can adjust the A and V distribution while mantaining the same W.

W, A, and V are instantaneous values. To measure over time, you typically see Watt-hours (Wh) and Amp-hours (Ah). Your utility bill usually specifies how many Wh you used (or kilowatts for 1000 Wh; kWh), while your battery will identify the amount of power it can store in terms of Ah at a given V. If you use fewer amps, then the battery will last longer.

Keep in mind, this can really screw up the power calculations if you get them wrong. For example, my 12V 100Ah DC battery is being converted to 120V AC power. If the AC uses a 1-amp load (like one server in the rack), then that's not 100 hours of battery; that's 10 hours. Why? 12V at 100Ah is 1200Wh. 1200Wh÷120V=10Ah, or 10 hours of power. (And with inverter's overhead and conversion loss, it's actually less.)

Parts and Parts

While I work with computers daily, I'm really a "software" specialist. Besides a few embedded systems, I don't do much with hardware. Moreover, the computer components that I deal with are typically low voltage DC (3.3V, 5V, or 12V and milliamps of power; it's hard to kill yourself if you briefly short out a 9V battery).

When it comes to high voltage, my electrical engineering friends all had the same advice:

1. Don't kill yourself.


2. Assume that all wires have enough power to kill you. Even when turned off.


3. When possible, over-spec the components. If you need 5 amps, get something that can handle 10 amps. If you need 12 gauge wire (12awg), then use 8awg (a thicker wire). If you need 2 hours of power, get something that can provide 4 hours of power. You can never go wrong by over-spec'ing the components. (Not exactly true, but it's a really good heuristic.)

For the last year, I've been using some Shelly plugs to monitor the energy consumption of my server rack. Every hour I take a reading and store it in a database. I also wrote a web interface that can display the real-time information and graph the hourly usage. (Every vertical bar is an hour, and every color is one day.)



The lower part of the rack hosts FotoForensics, Hacker Factor, and my other primary services. It usually consumes about 230W of power, or 2A. (It can fluctuate up to almost 300W during a reboot or high load, but those don't last long.) The upper rack is for the development systems, and uses around 180W. (180W at 120V is 1.5A.) That's right, the entire rack is usually consuming less than 4Ah of power at any given time.

For this solar experiment, I decided to initially only power the upper rack with solar. (If it turns out to be really successful, then I might add in the lower rack's power needs.)

The Bad Experiences

I had a few bad experiences while getting this to work. I chalk all of them up to the learning curve.

Problem #1: The Battery
Setting up the MPPT, inverter, and ATS was easy. The battery, on the other hand, was problematic. There are lots of batteries available and the prices range wildly. I went with a LiFePO4 "deep cycle" battery because they last longer than typical lead acid and lithium batteries and are designed for repeatedly powering up and draining. LiFePO4 also doesn't have the "toxic fumes" or "runaway heat" problems that the other batteries often have.

I found a LiFePO4 battery on Amazon that said it was UL-1973 certified. (That means for use with a solar project.) However when it arrived, it didn't say "UL 1973" anywhere on the battery or manuals. I then checked with Underwriter Labs web site. The battery was not listed. The model was not listed. The brand was listed, but none of their products had UL certifications. This is a knock-off forgery of a battery. If they lied about their certification, then I'm not going to trust the battery.

Amazon said that the vendor handles returns directly. My first request to the vendor was answered quickly with an unrelated response. I wrote to them: "I'd like to return the battery since it is not UL certified, as stated on your product description page." The reply? "The bluetooth battery needs to be charged before you can use it." (This battery doesn't even have bluetooth!)

My second request to the vendor received no response at all.

I told my credit card company. They stopped payment, sent an inquiry to the vendor, and gave them 15 days to respond. Two weeks later, with no response, I was refunded the costs. The day after the credit card issued the refund, the vendor reached out to me. After a short exchange, they paid to have the battery returned to them.

The second battery that I ordered, from a different vendor, had all of the certificates that they claimed.

Problem #2: The Inverter
The first inverter that I got looked right. However, when I connected it to the ATS, the wall outlet's circuit breaker immediately tripped. Okay, that's really bad. (But also, really good that the circuit breaker did its job and I didn't die.) It turns out, inverters above a certain wattage are required to have a "neutral-ground bond". The typical American three-prong outlet has a hot, neutral, and ground wire. The N-G bond means that neutral and ground are tied together. This is a required safety feature. Every home and office circuit has exactly one N-G bond. (It's in the home or building's circuit breaker panel.)

The four-poll (4P) ATS ties all grounds together while it switches the hot and neutrals. The problem: If the inverter and wall outlet both have a N-G bond, then it creates a grounding loop. (That's bad and immediately trips the circuit breaker.) For most inverters, this functionality is either not documented or poorly documented. Some inverters have a built-in N-G bond, some have a floating neutral (no bond) and are expected to be used with an ATS, and some have a switch to enable/disable the N-G bond.

My first inverter didn't mention the N-G bond and it couldn't be disabled. Fortunately, I was able to replace it with one that has a switch. With the N-G bond safely disabled, I can use it with the ATS without tripping the circuit breaker.

Keep this in mind when looking for an inverter. Most of the ones I looked at don't mention how they are bonded (or unbonded).

Problem #3: The ATS
I spent days tracking down this problem. The ATS output goes to a big UPS. This way, any transfer delays or phase issues are cleaned up before reaching the computers. When the inverter turned on, I would see a variety of different problems:

• Sometimes the UPS would run fine.


• Sometimes the UPS would scream about an input problem, but still run off the input power.


• Sometimes the UPS would not scream, and would slowly drain its internal battery while also using the inverter's power.


• Sometimes the UPS would scream and refuse to use the input power, preferring to run off the UPS battery.

The problem was incredibly inconsistent.

If I removed the ATS, then the UPS had no problem running off utility power. If I moved the electrical plug manually to the inverter (with the N-G bond enabled), it also ran without any problems.

Long story short: Most automatic transfer switches have a "direction". If primary is utility and backup is the generator (or solar), then it demands to be installed in that direction. For my configuration, I want a battery-priority ATS, but most ATSs (including mine) are utility-priority. You cannot just swap the inputs to the ATS and have it work. If, like me, you swap them, then the results become incredibly inconsistent and will lead you down the wrong debugging path.

My solution? Someday I'll purchase a smart switch that is battery-priority. In the meantime, I have a Shelly smart-plug monitoring the utility power. My DIY smart controller tells the Shelly plug to turn on or off utility power. When it turns off, the ATS immediately switches over to using the inverter's power. And when my DIY controller see that the solar battery is getting low, it turns the utility grid back on.

The added benefit for my method of turning on or off the utility power is that I can control the switching delay. The inverter takes a few seconds to start up. I have a 15-second timer between turning on the inverter (letting it power up and normalize) and turning off the utility power. This seems to help the UPS accept the transfer faster.

Problem #4: Over-spec'd Inverter
Remember that advice I got? Always over-spec the equipment? Well, that's not always a good idea. As it turns out, a bigger inverter requires more energy to run (18Wh for a 2000W inverter vs 12Wh for a 1000W inverter). It also has a worse conversion rate for a low load. (The inverter claims to have >92% conversion rate, meaning that the 100Ah battery should last for 92Ah. But with a light load, it may be closer to 80%.)

I'll stick with the inverter that I got, but I could probably have used the next smaller model.

Problem #5: The Roof
I wanted to put the solar panels on the roof. I really thought this was going to be the easiest part. Boy, was I wrong.

There are federal, municipal, and local building requirements, and that means getting a permit. The city requires a formal report from a licensed structural engineer to testify that the roof can hold the solar panels. Keep in mind, I'm talking about two panels that weigh 14lbs (6kg) each. The inspector who goes up on the roof weighs more. If a big bird lands on my roof (we have huge Canadian geese), it weighs more. We get snow in the winter and the snow weighs more!

Unfortunately, the city made it clear that there is no waiver. I had earmarked $1000 for everything, from the panels to the battery, inverter, wires, fuses, mounting brackets, etc. I got quotes from multiple structural engineers -- they all wanted around $500. (There goes my budget!) And that's before paying for the permit. In effect, the project was no longer financially viable.

Fortunately, I found a workaround: awnings. The city says that you don't need a permit for awnings if they (1) are attached to an exterior wall, (2) require no external support, and (3) stick out less than 54 inches. My solar panels are mounted at an angle and act as an awning that sticks out 16 inches. (The mounts are so sturdy that I think they can hold my body weight.) No permit needed.

The awnings turned out to be great! They receive direct sunlight starting an hour after sunrise and it lasts until about 1pm in the summer. (It should get even more in the winter.) They continue generating power from ambient lighting until an hour before sundown. This is as good as having them on the roof!

The Scariest Part

High voltage scares me. (That's probably a healthy fear.) Connecting cables to the powered-off system doesn't bother me. But connecting wires to the big battery is dangerous.

Using rubber-gripped tools, I attached one cable. However, when I tried to connect the other cable, there was a big spark. It's a 100Ah battery, so that's expected. But it still scared the donuts out of me! I stopped all work and ordered some rubber electrical gloves. (Get rubber or nitrile, and make sure they are class 00 or higher.)

Along with the gloves, I ordered a huge on/off switch. This isn't your typical light switch. This monster can handle 24V at 275 amps! (It's good to over-spec.)

I connected the MPPT and inverter to one side of the switch. An 8awg cable that can handle 50 amps connects to the battery's negative pole. (Since the MPPT and inverter are both limited to 20 amps, the 50 amp cable shouldn't be a problem.)

With the gloves on, the switch powered off, and rubber-gripped tools in hand, I connected the switch to the battery. No zap or spark at all. Turning the switch on is easy and there is no spark or pop. This is the right way to do it.

Expected Savings

Without the solar project (just using the utility power), the server rack costs me about $35 per month in electricity to run.

I've been running some tests on the solar project's performance, and am very happy with the results.

Under theoretically ideal conditions, two 100W panels should generate a maximum of 200W. However, between power conversion loss, loss from cabling, and other factors, this theoretical maximum never happens. I was told to be happy if it generated a maximum of 150W. Well, I'm very happy because I've measured daily maximums between 170W and 180W received at the MPPT.

Fort Collins gets over 300 sunny days a year, so clear skies are the norm. With clear skies, the battery starts charging about 30 minutes after sunrise and gets direct (optimal) sunlight between 9am and 1pm. It generates an incredible amount of power -- the inverter drains the battery slower than the panels can charge it. For the rest of the afternoon, it slowly charges up the battery through indirect ambient light. The net result? It can run the upper half of the rack for over 10 hours.

This kind of makes sense:

• The battery usually starts the day at about 50% capacity. It charges to 90% in under 2 hours of direct daylight.


• In theory, I run the battery from 90% down to 20%. In practice, the battery usually hits 100% charged during the morning because if charges faster than it drains. It doesn't start running below 100% until the afternoon. (That's wasted power! I need to turn on more computers!)


• I'm only using the upper rack right now (1.5Ah, 180Wh). The inverter consumes another 18W, so call it 200W of power. Assume a fully-charged 1200Wh battery with 920Wh available, draining at a rate of 200Wh. It should last about 4.5 hours. If I power the entire rack, it will be closer to 2 hours. And in either case, that countdown only starts when it's running off of battery in the late afternoon.

We had one dark and stormy day, and one very overcast day so far. In both instances, it took most of the morning to charge the battery, but it still managed to run the upper rack for a few hours. Fort Collins has "surge pricing" for electricity. In the summer, than means 2pm to 7pm has the most expensive power (about 3x more than non-surge times). Fortunately, the batteries keep the rack running during much of that expensive period.

I'm aiming to use the battery during the surge pricing period. If I ran the numbers correctly, the server might reduce my $35/month cost by $20-$25/month. At that rate, it will pay off the $1000 investment in under 4.5 years. If we have a lot of bad weather, then it might end up being 5 years. The batteries and panels will need to be replaced in 8-10 years, so as long as it pays off before then, I'll be in the profit range.

As self-paced learning goes, I don't recommend high voltage as an introductory project. Having said that, I really feel like I've learned a lot from this experiment. And who knows? Maybe next time I'll try wind power. Fort Collins has lots of windy days!