Goldfinches are one of the most brightly colored common birds in the DC area. If you saw one in a tropical area, you might be transfixed...but here, it's just a goldfinch. Well, stop and let yourself be transfixed anyway. They're lovely birds.
Unlike most birds, goldfinches rely almost entirely on seeds; they don't seek out insects for food at all. In fact, you could call them "thistle birds" -- the first part of the Latin name Carduelis tristis comes from Carduus, a genus of thistle found in Europe and Asia. (The species name, tristis, comes from the Latin for "sad," which makes no sense to me - anyone?)
Besides looking for the distinctive yellow markings, you can learn to listen for the song of the goldfinch. This video has a good example:
In the wild: Look for meadows on edges of woods. Goldfinches will feed on the seeds of the sun-loving flowers in the meadow (where you'll be able to see them better), but nest in the trees and shrubs.
In your yard: You can purchase birdfeeders specially designed for goldfinches, and fill them with thistle or nyjer seed. Squirrels seem less interested in these tiny seeds than in a typical birdseed mix, but be careful -- those plants may start growing where you don't want them from spilled seed. Goldfinches will also eat sunflower seeds out of a more typical birdseed. Or, just plant yourself a goldfinch garden with some of the native flowers listed above.