I have a recollection of listening to a classic recording of this opera, I suspect one of a number of opera CDs inherited from a beloved uncle, and finding it enjoyable. There's certainly a lot of beautiful music in the piece, and I'm not sorry to have heard it live, but it is also not a mystery to me that it has largely fallen out of the repertoire.
The first problem lies with the way Saint-Saens structures the piece. The drama has an episodic feel, and tension both musically and dramatically has a tendency to drop. The balance between ensemble spectacle and exploration of the principal characters doesn't seem quite right. Most seriously there appears to be a glaring omission to stage a key plot point - that is so far as I could judge Dalila never does get the secret of Samson's power out of him at the climax of Act Two thus making it inexplicable that he can't escape from the mob.