AMMRL: Follow up: Spikes in FID on Gemini 2000/300MHz
jtomasze (jtomasze@indiana.edu)
Thu, 10 Jan 2008 15:33:48 -0500
Fellow NMRtists,
I'd just like to give a belated, but no less sincere, thanks to:
Edwin, Sue, David, Charles, Bill, Steven, Thomas D., Jerry H., Martha,
Thomas S., Ken, Dave, Robert, Joseph, the other Jerry H., and Christian,
for all the suggestions and trouble shooting advice. After checking and
cleaning many connections and confirming the PS and masterclock were
clean, we had it narrowed it down to either the CPU, ADC, or Enet/Comm
board. Working with Jerry Hatvany from Triangle Analytical, we got
trial boards for testing and it turned out to be the Enet board. One
quick swap, and our little 300 has been purring like a kitten since.
Original post:
> We have another hardware problem that I would like to throw out for
> comments if I may.
> On our Gemini 2000/300MHz we started getting random spikes in the
> FIDs a few weeks ago. Then it was maybe one bad FID in a set of 16.
> The frequency has been increasing and some bad scans seem to abort the
> experiment and lock up the console requiring a "su acqproc" or
> reboot. In the MasterLog file we see "Data Hal returned Exceeds
> buffer size Exp ID = #.CP-Exx" where # is the experiment number (since
> the last reboot), and xx is some numeric value. The problem is
> independent of the nucleus being observed, but there does seem to be
> some temperature dependence. Last week, in the mornings when the room
> temp was ~19.0C or above we had problems, but in the afternoons, if
> the temp dropped below 18.5C we could get some experiments to
> complete. Today, even with a room temp of 18.0C and an extra fan in
> the console, I'm lucky to get a single scan experiment done and they
> all have the spikes. The temperature dependence seems odd though,
> because neither the console nor the CPU are hot, or even warm for that
> matter.
> I've searched the archives but don't see any similar problem
> descriptions. We were thinking maybe a dying CPU or memory chip, or
> perhaps something on the ADC board. Anything else we should
> investigate? If we start getting boards to swap in for testing from
> Varian, does anyone have any thoughts on which board(s) to try first?
> Many thanks in advance.
Below are all the replies for the archives...
> Do you have another probe where you can rule out the probe for
> arching? But then that would not explain the hault in the acquisition
> unless you are exceeding some number in the Acquisition range in the
> ADC board.
>
> I do remember having similar problems on an inova400. The problem was
> both preamplifier and probe.
>
> Probe testing: 90 array @ a the standadr power then at 6 or 12dB lower.
>
> Preamp testing; swapping with another preamp or injecting some Rf and
> measuring the output.
> We've just replaced our gemini but from past experience the problems
> you describe may be related to your OBS/REC board. We had two issues.
> One was simply the board contacts. I would regularly need to turn off
> the instrument, remove the board, clean the contacts, reseat it and
> wiggle it about a bit for good measure. This was always necessary if
> our air-con went down (the outside temp here is usually quite
> different from lab temp : ) ). The second problem was more difficult.
> The now very difficult to obtain attenuator relays kept fritzing on
> us. You can test for this experimentally by varying your receiver
> gain, if it's one of the relays you shouldn't observe the problem at
> all levels of attenuation. I'd send you our old board however the
> relays on it have been replaced several times with salvaged parts
> already (I put the grants in to replace it because I knew we couldn't
> keep up the bandaid cures forever).
> I don't know if it applies to the Gemini systems, but on Inova consoles which are nominally 220 volts, you can get by with 208 but all kinds of bad stuff happens if it goes down even as low as 204. Although this isn't the normal season for brownouts, it might be worth checking into--also your power supplies in general.
> I would try the following, albeit using a lot of care (esp. with
> static protection) when handling the boards. Remove one board at a
> time, starting with the one considered most likely (CPU or ADC).
> Check that any removable chips are firmly seated in their sockets.
> Clean the backplane pins and put back in the console. If you think
> pulling boards is more risky than powering up, go ahead after each
> board and power up to see if the problem is fixed (or at least reduced
> in frequency of occurrence). If you think power ups are more risky (I
> had an old MSL that hated power-ups; never had a Gemini), pull all the
> other boards and do similar, then power up. Perhaps you can fix the
> problem this way.
>
> Also, check your cables that all the connectors (especially) are
> solid. If any look suspicious, replace them.
>
> A bit of a shot in the dark, although we have fixed problems, and I've
> heard similar stories from other labs (and esp. on older consoles),
> using this procedure.
> First thing I would do is put a scope on the power supplies to see if any
> spikes or noise is visible, especially on DC outputs.
> Check your power supplies. I suspect you'll find a bunch of AC ripple on
> a 5 volt supply in the acquisition console. You may only have to replace a
> filter capacitor....
> I would compare the real and imaginary fids to ensure they both are the
> same amplitude.
>
> The lock display is routed through the ADC's for graphics display on the
> Sun. Does the system hang up when
> you display the lock over long periods of time? This would indicate an
> ADC problem. Try reseating the board.
>
> Finally, monitor the 5v & 12V power supplies to the console computer
> backplane. The AC ripple should be lower
> than a few millivolts as measured with a fluke volt meter in RMS mode.
> Did you make sure that the hidden cooling fans under the digital
> cardcage are working? Also check ripple on the main 5 volt supply.
> There is a hidden fan which blows right on that supply only
> accessible if you pull the power supply out that we've seen go out
> quite a bit lately.
> I had something similar on a Bruker several years ago. My problem was a loose
> connector to the ADC board. I knocked it while setting up a different nuclei.
> I would start at this board.
> Have you examined the FIDs and/or the synthesizer output? Some time
> ago I experienced intermittent spectral glitches such as spikes, and
> ultimately---via observing "holes" (one or more points of zero
> magnitude) in the FIDs---traced this back to a failing/failed PTS
> unit. (And I returned the synthesizers to PTS for repair; they
> repaired one under their own warranty even though the Varian warranty
> had expired! How's that for good old-fashioned service?)
> Though I have not seen this but I've heard arcing taking place in the
> probe will produce random spikes in the fid. As a test you could run a 1d
> proton experiment. Lower the transmit power, give yourself a short pulse
> and see if the frequency and/or amplitude of the apikes have reduced. If
> the spikes are reduced this maybe the problem. If this turns out to be
> the problem before installing a new probe and frying that one, you should
> ensure your transmit powers are correct.
> We've been getting 1H spikes on a 400MHz Bruker when a nearby
> Varian 400 is doing waltz decoupling, so I guess you can't
> totally discount external rf. But I doubt thats your
> problem. If the spikes are separated by 60Hz, there is
> usually a power supply involved. You could try unplugging
> the shims or the lock to see if they disappear.
> Look at FID's. Single point spikes show as DC offsets somewhere along the
> FID. A DC offset over the entire range of a single scan is the tof spike in
> the center. If that is wildly off scale you can get that. So one possible
> place to look is an intermittent offset adjustment circuit. Maybe bad
> transistor or op-amp.
>
> Make sure the fans are working. There is a warning buzzer for the
> powersupply fans which you hear momentarily on powerup ... but I don't think
> there are alarms for the others. There are two fans directly below the
> upper cards. Get out the schematics for the ADC and ID the location of the
> IC's. When it fails quickly shutdown and check the temp with your fingers
> of each IC. The hot once will be the bad ones and likely a more significant
> bit. The inverse check is some electronics troubleshooting coolant spray
> (non-flammable) which you can spray in there intermittently to narrow it to
> a card. The thing with these failures is each Hi-temp excursion damages the
> device more and it eventually works no more.
>
> I don't remember if they are socketed. If they are socketed you can
> CAREFULLY (minimal prying -proper tools- static grounded) remove them and
> swap LSB with MSB to see if the problem minimizes. If you have electronic
> repair facilities with solder removal stations, it's best to replace them
> all if you have more. Same age devices ... You'll likely have to bid out
> the parts which will be obsolete. You'll get dozens of replies ... mostly
> fake of people who say they have it but only intend to search for it. So
> make sure they have them in stock for delivery today.
>
> Bench repair at Varian is ~$500/hr the swap out price might be astronomical
> (55% retail) so it could be worth it. If you can wait for a Varian visit by
> an Engineer who is coming for a primary customer on campus you can save
> travel costs and if he brings all the parts he needs the time cost could be
> as little as 2hrs.
>
> It may have nothing to do with your ADC. Check the 300Mhz filter in the LH
> magnet leg. I had one short to SMOKE and afterward it did all sorts of
> crazy things. This included HAL errors even on other nuclei because
> fiberglass is an excellent insulator but fiberglass composite charcoal is an
> excellent conductor and it loads the pulse programmer. Open the door on the
> magnet leg and you will smell it if it has failed. (55% x $1000 with trade).
> You'll want to hipot your wires afterward too.
>
> The bad news is that it can melt your wires inside back to your AMT and
> cause other issues. That would be an arcing issue in the magnet leg. Turn
> your transmitter power way down by a 25W power filter or console control and
> see if the spikes subside. If it is that little filter the tuning to the
> probe will be way off from before.
>
> If it's your CPU board you are basically hosed. You must have it bench
> repaired, I heard they were out of replacements. Or you can look on the
> used market.
> I would look at the DC power supplies. What is the frequency of the
> noise spikes? If it is 60Hz, (or a multiple of 60Hz, like 120Hz) it
> usually points to a power supply.
>
> A DC power problem would explain all of your symptoms, even your
> temperature problem.
> Considering that all nuclei are affected, it sounds like a power supply
> issue. I would check all power supply voltages first, then suspect the A/D
> conversion.
> I have seen such problems on an old AC BRUKER spectrometer when the numeric processor was dying.
> Perhaps your Geminin is of the same vintage.
--
John W. Tomaszewski, Ph.D.
IU, Dept. of Chemistry
800 E. Kirkwood Ave.
Bloomington, IN 47405-7102
(812) 855-4478
jtomasze@indiana.edu