Italy already gave the Pope concern. The Ghibellines, or Imperialists, had grown powerful in the Pope's absence, and their chief leader, Matteo Visconti of Milan, a ruthless and exacting ruler, was "Imperial Vicar" in the country. When Visconti, in defiance of the Pope's commands, gave aid to the Ghibellines of Genoa, John, who claimed to represent the Empire during the "vacancy," withdrew his title of Vicar and awarded it to Robert of Naples. Robert went to consult John at Avignon, and a campaign followed. Cardinal Bertrand de Poyet--who was, says Petrarch, so much like John "in face and ferocity"[261] that one could easily credit the rumour that he was John's son--was sent to direct the Papal cause and to denounce the Viscontis to the Inquisition. Matteo was found guilty of heresy (or contumacious refusal to abandon the title of Vicar), and he and his son were charged with oppression of the clergy (which is plausible enough) and with a quaint and amusing mixture of magic and other devilry.[262] Possibly John relied more confidently on the troops of Philip of Valois and Henry of Austria, whom he successively summoned to Italy; but they retired almost without a blow. Matteo repented and died, but his sons and their associates continued the war.
At this juncture Louis conquered Frederic and sent word to the Legate to keep his troops out of imperial territory. When the Legate refused, he joined the Ghibellines and drew from John a vigorous denunciation. He was to abandon the "heretics" and come to Avignon for the examination of his claim to the Empire. Louis, retorting (under the inspiration of the friars) that there were heretics at Avignon as well as in Italy, went his way, and John turned to France. Charles the Fair, the new King, had discovered that, when Clement V. had authorized his marriage with Blanche of Burgundy, a remote godmothership had been overlooked, and he was in the painful position of living with one to whom he was not validly married. John declared the marriage void, allowed Charles to marry another lady, and was soon in conference with Charles and with Robert of Naples. Germany took alarm at this plain hint of an intention to make Charles Emperor; the Italian spiritual war upon the Pope was vigorously repeated in that country, and the Diet of Ratisbon rejected John's authority and called for a General Council.