In 1963 Sinatra invested $55,000 in the Berkshire Downs Racetrack in western Massachusetts. He and Dean Martin were named directors of the track. Rumours abounded, and were often repeated in the FBI files, that Sinatra was a front for the real owners of the racetrack: Raymond Patriarca and Gaetano "Three-finger Brown" Lucchese. The track went bankrupt in 1965, but the deal came back to haunt Sinatra years later.

In 1972 the House Select Committee on Crime prepared a subpoena for Sinatra to question him about the Berkshire Downs Racetrack. A phone call from Sen. John Tunney of California, for whom Sinatra had raised $160,000 during his re-election campaign, stopped the subpoena. Tunney informed the committee, after talking with Mickey Rudin, Sinatra’s lawyer, that Sinatra would be glad to appear before them if he were invited. The committee obligingly invited Sinatra to appear on June 4, 1972. Instead Sinatra flew to England.

Incensed the committee prepared a second subpoena and ordered U.S. marshals to stand by at all ports of entry to issue the subpoena as soon as Sinatra re-entered the United States. Sinatra called in some favours and after telephone calls from Vice President Spiro Agnew, a close friend of Sinatra, the second subpoena was cancelled and a new invitation was issued for July 18. In addition the committee agreed to limit their questions to the Berkshire Downs Racetrack. Sinatra took a belligerent attitude with the committee when he appeared. He also apparently lied several times. For example:
Q: Tell us about the first contact you had with anyone in relation to Berkshire Downs?
A: The first and only contact I had was with a man named Sal Rizzo.
Q: How did you know Mr. Rizzo?
A: I met him.
Q: How?
A: I can’t remember where or how, but I met him and we got to talking about it and I liked his idea for the investment.

Later Charles Carson, the racetrack controller, would testify that Rizzo and Sinatra were childhood friends. Rizzo had told him that, "I have known Sinatra since New Jersey. I was a neighbor of his and I knew the whole family."

When Rizzo appeared in front of the committee he took the Fifth Amendment on 34 of the 46 questions he was asked, refusing to say anything about his relationship with Sinatra, or even if he had lived in New Jersey. Testimony Rizzo had given to the Florida Beverage Control Commission (FBCC) in 1968 was read into the record in which Rizzo testified that Sinatra had received money from Berkshire Downs and that Rizzo had known Sinatra for "fifteen to twenty years."

Commenting on Sinatra and Rizzo’s testimony, the committee counsel said, "It appears … that Mr. Sinatra’s testimony … was false, or the testimony of Mr. Rizzo before the [FBCC] was false. In either case one of the gentleman has committed perjury." Raymond Patriarca was brought before the committee from the Atlanta Federal Penitentiary, where he was serving 10 years for murder conspiracy. Asked if he knew Sinatra, the boss of New England organized crime said:
A: I never met Sinatra personally. I seen him on television and at the moving pictures.
Q: Did you ever have any business dealings with him?
A: No, sir.
Q: Did you ever purchase stock from him?
A: No, sir.
Q: Anybody on your behalf do it?
A: I claim my Fifth Amendment privilege.
Q: Do you have any knowledge that anyone associated with you had any business dealings with Sinatra?
A: I claim my Fifth Amendment privilege.

It may be true that Sinatra never met Patriarca personally, but Mickey Rudin, Sinatra’s attorney, had earlier acknowledged his client’s friendship with Gaetano Lucchese, and Patriarca’s testimony left the question of his role in Berkshire Downs in question.

The committee members, apparently intimidated by Sinatra, virtually apologized to him before they were through. Representative Charles Rangel, of New York, who was a beneficiary of Sinatra’s fundraising, told him, "You’re still Chairman of the Board."

Sinatra kept up his attack on the committee after his 95-minute appearance. Shortly afterward he sent the committee a bill for $18,750 in expenses. It was never paid, but in July The New York Times published a commentary bearing Sinatra’s byline wherein Sinatra claimed that the committee had invaded his privacy in order to gain publicity during an election year. The committee backed down and no further action in the Berkshire Downs case was taken.

President Nixon, happy over the embarrassment of the congressional committee, made a personal phone call of congratulations to Sinatra. This personal contact from the President, and Sinatra’s friendship with Vice-President Agnew, led to Sinatra’s support of Nixon’s re-election in 1972.

Sinatra’s changing political parties helped him strengthen his political influence in the next two decades. It also led to his most blatant illegal actions in support of his organized-crime friends.