Lawmakers offer little hope oversight panel will be removed