Picture the scene. You fly out for an integrated factory acceptance test, walk onto the floor, and the first thing you check isn’t the tag list or the custom logic. It’s whether the two systems can see each other at all: whether the OEM controller and the OPC server are actually talking. If that hasn’t happened before you walk in, you already know how the week is going to go.
It sounds like a small thing to check first. It isn’t. One of our engineers arrived on site not long ago, installed the software, and could not bring up communication no matter what he tried. The reason turned out to be almost impossible to guess from a spec sheet: the hard cable between the APC server and the DCS had never been physically connected. Another team walked into a kickoff meeting where every readiness item, hardware on site, communication tested, and prior action items closed had reportedly been confirmed weeks earlier. Yet, none of it had actually been done.
That’s the scene the three of us — Madhur Bedre, Nick Brown, and Amlan Mahamallik — kept coming back to when we sat down recently to compare fifteen years of IFAT experience. The IFAT has always had pain points, and it always will. Despite improvements, the issue that remains is where those pain points come from.
What Used To Wreck an IFAT?
Ask anyone who was running these tests fifteen years ago and you’ll hear the same word: OPC. A single week could disappear into it. The protocol was built in the 1990s, and it behaved the way things from that era tend to: brittle. A changed password, an altered authentication level, someone moving a group on the server side, or the way a network switch happened to be configured could freeze communication, often for no reason anyone could name. Troubleshooting turned into a kitchen sink approach: try something, see what breaks next, then try something else. There were entire companies whose business model was helping other people get OPC working.
Installation had its own version of the same problem. Individual installers were run by hand, one at a time, across multiple days or weeks, and the process left plenty of room for mistakes that wouldn’t surface until testing began. Sometimes the culprit wasn’t the installation at all, it was the antivirus software, quietly locking files and preventing the OEM software from running.
So What Fixed It?
Sequencing replaced the manual, one-by-one installation process, cutting what used to take days down to a fraction of that. OPC UA closed most of the remaining reliability gap that older DCOM-based communication left open. Antivirus and endpoint security are still set by the customer’s IT department, but as virtualization matured, the file-locking and continuous-backup issues that used to eat entire days largely disappeared.
Error handling changed too, and this is worth sitting with for a second. A DCOM or OPC fault used to send an engineer hunting through documentation for a matching string. Now that string goes into an AI assistant, comes back with a plausible direction to try, and gets handed to the DCS engineer as something concrete he’ll recognize. One of us fixed an IFAT issue that way just weeks before this conversation.
None of this means OPC and DCOM became foolproof overnight. It means the failure modes that used to eat a week now get resolved in an afternoon. The needle moved. We just face a different spread of issues now.
If the Tech Got Better, Why Do IFATs Still Go Sideways?
Here’s a question worth asking about your own last IFAT: did the client confirm readiness, or did they demonstrate it?
Every failure we still see traces back to a gap between what someone said was done and what was actually completed. The pattern is familiar. The client confirms the hardware is on site, communication has been tested, and the kickoff action items are closed. The team shows up, and it turns out that none of that, aside from maybe the hardware delivery, actually happened. We’ve been asking for these confirmations for years, which is exactly why a formal Pre-IFAT meeting now exists, dedicated to walking through everything that has to be finished before we arrive. What the process still lacks is evidence. A stated confirmation and a demonstrated one are not the same thing. We’re now looking at requiring visual proof — photos of a running communication link, or a short demo call showing the OEM logic live on the vendor’s hardware — before we commit to travel and schedule a date.
The same gap shows up in a quieter form during the test itself. Ask a question to get information you can act on, and the answer that comes back isn’t always coming from someone who actually has that information. Configurations built during the IFAT sometimes get rejected months later on site, once the person who should have been in the room from the start finally weighs in. Checklists have the same weakness: working through a checklist and actually achieving the result it’s supposed to verify are two different activities; it’s easy to complete the first without the second.
The gap survives past the IFAT, too. A report goes out assigning responsibility, the hardware ships, and the DCS logic touching the APC system gets reviewed and modified behind the plant’s firewall, out of view. By the time we return for commissioning, the configuration on the ground no longer matches what we tested at the vendor site. Scheduling pre-commissioning closer to startup narrows that gap, though it doesn’t close it. What actually seems to matter most is how the customer treats the FAT and IFAT internally. Where they’re managed as real project milestones, rework drops noticeably. Where they’re treated as flexible checkpoints, activities stay loosely defined even after the hardware has shipped, and that’s exactly where the hidden changes creep in.
Who's Actually at Fault When Two Systems Disagree?
Intermittent faults, errant values, tags that stop reading, and logic that fails without a pattern are some of the hardest problems to catch inside a fixed test window. They’re also the fastest way for a technical disagreement to turn into a standoff over who’s at fault.
On one project, we saw sporadic communication loss between a major DCS vendor’s redundant OPC servers and our own, a problem that carried through the IFAT and into commissioning and startup. Both sides insisted their system was fine. Oddly enough, both were telling the truth. What actually broke the impasse was a log file: every point sent, response received, and read and write that failed. After being handed over in full, the DCS side checked it against their own records and traced the cause to their redundancy layer. It took weeks of shared logs to get there, but the log, not the argument, is what settled it.
On a different project site, we had values arriving at the OEM console that looked almost random: 80°C on the DCS screen, roughly 1,000°C on our side, 1,200°C a few seconds later, then back down near 300°C. Tag names matched. The vendor understandably declined ownership, since their own screen showed the correct number. We reproduced the same behavior across several independent OPC explorers, and once that evidence was in hand, the vendor engaged immediately. The demonstration happened at ten in the morning and the fix landed by noon. Getting to that demonstration, though, took days.
As Madhur put it during our conversation: “Neither of these cases were really about OPC. They were about ownership.”
Before a technical issue can become a solvable one, everyone in the room has to agree it’s one shared problem, not two competing ones. It isn’t the DCS’s issue and it isn’t the OEM’s. Onsite, it’s a team problem, and it needs a team’s worth of problem solving.
Vendor Site or Plant Floor? And Should You Ever Do It Remote?
This one comes up on nearly every project, so let’s settle it as clearly as we can.
The technical objective of an IFAT is straightforward: combine the OEM software with the DCS vendor’s software, prove out communication and tag verification in a factory setting, and get most of the way through installation and custom logic before the hardware ships. The second objective gets far less attention and matters just as much: the people. Until the IFAT, we haven’t actually worked with this particular customer, this DCS vendor, or these engineering vendors. It’s the first time everyone is in a room negotiating priorities in real time, and how that week goes is a reasonably good preview of how the rest of the project will go.
The strongest argument for the vendor site is availability. The resources are already there, and the vendor is highly motivated, because once the hardware ships, liability and exposure shifts, and a full vendor team can shrink down to a single engineer. Every IFAT surfaces something that needs working through, and the vendor site is simply the best place to do that work. Its drawback: the system in that lab has usually changed since the last time it was configured, right down to the logins, so a fair amount of setup has to be redone from scratch.
Running the IFAT at the plant unit has its own trade-offs. The process control network there is the real network, not a stand-in that comes apart once the hardware moves. You can settle exactly where the APC screen sits on the control panel, and operators can take part directly, which meaningfully raises the odds they’ll actually use the system once it’s commissioned. The tradeoff is vendor support. Without it readily available, a routine week can turn into a multi-month communication project if the onsite engineer runs into something unfamiliar.
And remote? We’ve done it. One fully remote IFAT in 2020, with a great deal of upfront planning, went better than the circumstances suggested it might. Vendors opened up access they never would have on a live plant, and both sides could pull in specialists on demand. What suffered was accountability: the customer wasn’t physically present to see decisions being made, and those decisions didn’t always make it back to them. A more recent remote IFAT took two and a half days to complete work that would have taken about half a day in person, because everything had to be explained over a call, and the DCS engineers weren’t always present or clear on what was being asked of them. Remote doesn’t invent new problems. It multiplies the ones you already have.
What Does Skipping It Actually Cost You?
Control loop testing doesn’t disappear if it doesn’t get done properly at the IFAT. It still has to happen before startup, except now you have days instead of months, and everyone around you is under the same pressure to get the unit running. That’s exactly when conflict tends to spike.
We describe it internally as fixing a leaking ship. A large number of accountable parties work together so that the ship floats once it’s back in the water. In the dockyard, there’s time to get it right. Once it’s in the water, there’s no easy way to plug a hole.
What Would You Tell a New Engineer?
We’ve spent years writing down procedures, troubleshooting guides, and documentation that used to live only in one engineer’s head. What resists documentation is the part of the job that doesn’t reduce to a checklist. Onsite, you’re negotiating constantly: with the back office to get your work prioritized, the customer for support and resources, and the vendor over what gets done first, all while continuing to solve the technical problem in front of you. Ask any of us what we’d tell a new engineer before their first IFAT and you’ll hear a version of the same line: work closely with the DCS engineer. He knows more about his own system than you ever will, and he’s the one best positioned to help you solve your specific problem.
The procedures will probably never be finished, and that’s fine. Every IFAT hits fifteen roadblocks, and we write down how we get past each one. The next IFAT hands us ten completely different roadblocks. Slowly, the coverage grows.
So, if you’re planning your next IFAT, here’s the short version of everything above: don’t just ask if it’s ready, ask to see it. Assume the intermittent fault is a shared problem before it becomes a shared argument. And remember that the technology got easier so the two of you could spend more time on the part that never got easier — working with the people in the room. That’s not a footnote to a successful IFAT. It’s the whole job.
