In February of 1969, during its fourth year in existence, the Sumter Artist Guild decided to establish a local gallery in which they could display their work as well as host exhibits from artists outside the community.

After over a year of planning and fund raising, the Sumter Gallery of Art opened its doors on January 5th, 1970. The Gallery was first housed in the Old Carnegie Library on Liberty Street, with the school district agreeing to lease the building to the Gallery for a fee of $1 per year. The opening exhibit featured works from the South Carolina State Arts Commission with assistance provided also by this group.

In 1977 the Gallery moved to a new location, an antebellum Greek Revival Style home on North Main Street. The home was built in 1850 by Dr. Anthony White. His grand-daughter, Elizabeth White, became one of Sumter's most talented artists and went on to gain national and international recognition. Upon her death in 1976, Miss White requested that her home be made "available to the public in a manner that will promote the arts." The house, which is on the National Register for Historic Places, remained the Gallery's home for the next 26 years.

In the mid-1980's the County of Sumter renovated a portion of the former Edmunds High School to a center for the performing arts, Patriot Hall. At this time it was suggested that the science wing adjacent to Patriot Hall be converted into a new home for the Gallery of Art.

On February 23rd, 2003 the Sumter Gallery of Art opened its doors to the public in the newly renovated 24,000 square foot facility at the Sumter County Cultural Center. With three exhibit galleries downstairs and seven classrooms upstairs, the Gallery is able to offer Sumter and surrounding communities the finest in art exhibits and educational opportunities.